Netbeans 8.1 “No tests executed”

Recently I’ve started to learn and practise unit testing in Java using JUnit to improve my productivity and confidence in my own code. Netbeans is quite useful for tests too, because it automates many things while writing and running the tests. However today I noticed something weird: I wrote some tests and I could run them just fine by themselves from the projects widget using the context menuitem “test file”, however when I wanted to “test project” from the run menu, Netbeans said “No tests executed” in the test window. I started digging and unfortunately I couldn’t find anything useful. However I noticed that in all the examples the test classes were called XYZTest, while my test classes are were named TestXYZ. I tried to rename then and guess what? Renaming them solved the problem. So to sum my experiences up:

If you want Netbeans to find and run your test classes name them like this: XYZTest. Where XYZ can be of course an arbitrary string that is legal in a class name in Java.

Advertisements

SocketTimeoutException in servlet running in Tomcat 7 behind an Nginx 1.10.3 reverse proxy

We have a digital object repository called DSpace at work, and we use the SWORDv2 protocol to deposit digital object into it. DSpace GUI and it’s SWORDv2 endpoint runs as servlets in a Tomcat container, and it’s all behind Nginx acting as a reverse proxy.

The other day one of my co-workers wanted to deposit a larger digital object package ( 8 GB ) into the repository, but unfortunately it failed because the servlet kept throwing SocketTimeoutException while it was reading the data being deposited, so I had to investigate and solve the problem.

java.net.SocketTimeoutException: Read timed out
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
java.net.SocketInputStream.read(SocketInputStream.java:170)
java.net.SocketInputStream.read(SocketInputStream.java:141)
org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:535)
org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504)
org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:566)
org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:137)
org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.java:339)
org.apache.coyote.Request.doRead(Request.java:438)
org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290)
org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:449)
org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:315)
org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:167)
org.swordapp.server.SwordAPIEndpoint.storeAndCheckBinary(SwordAPIEndpoint.java:197)
org.swordapp.server.SwordAPIEndpoint.addDepositPropertiesFromBinary(SwordAPIEndpoint.java:388)
org.swordapp.server.CollectionAPI.post(CollectionAPI.java:160)
org.swordapp.server.servlets.CollectionServletDefault.doPost(CollectionServletDefault.java:48)
javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

I read the Tomcat and DSpace logs but it revealed nothing. I noticed that DSpace had some interrupted deposits in it’s upload directory. All of the files were of size 2 GB, which was suspicious but I couldn’t figure out why at first, because I couldn’t see and find any limit that would explain why it should die at just 2 gigs.

I am not an Nginx expert, but I enabled debug logging and started reading logs. Unfortunately at first sight it didn’t reveal anything, I saw no errors, only that Tomcat returned 500 while depositing, that’s when the SocketTimeoutException was raised. However some lines caught my attention anyways.

2018/07/18 09:27:55 [debug] 4273#4273: *1 sendfile: @0 2147479552
2018/07/18 09:27:55 [debug] 4273#4273: *1 sendfile: 2147479552 of 2147479552 @0

That big integer was quite suspicous, and after doing some simple math I figured that 2147479552 twice divided by 1024 is 2048. Which means this could be a byte count. This made me start thinking. After sending this much data and some wait Tomcat sent 500 with that exception, so I figured it’s worth looking into. I started digging in Nginx’s source code and found a comment block and a constant below it:

/*
* On Linux up to 2.4.21 sendfile() (syscall #187) works with 32-bit
* offsets only, and the including <sys/sendfile.h> breaks the compiling,
* if off_t is 64 bit wide. So we use own sendfile() definition, where offset
* parameter is int32_t, and use sendfile() for the file parts below 2G only,
* see src/os/unix/ngx_linux_config.h
*
* Linux 2.4.21 has the new sendfile64() syscall #239.
*
* On Linux up to 2.6.16 sendfile() does not allow to pass the count parameter
* more than 2G-1 bytes even on 64-bit platforms: it returns EINVAL,
* so we limit it to 2G-1 bytes.
*/

#define NGX_SENDFILE_MAXSIZE 2147483647L

After some further digging I realized that this sendfile() call is the default network I/O implementation of Nginx, but it can be turned off by setting

sendfile off;

in the http scope of the Nginx config file. As I suspected this solved the problem, and we could deposit the packages without problems. Now as a short summary here’s what this is about and what happened:

sendfile() is an I/O call that transfers data between file descriptors without having to first read the data into RAM, therefore it’s faster than the traditional solution of reading from the source, storing in RAM then writing to the destination. This is by default enabled in Nginx and this is what among other solutions makes Nginx a fast web server. However it has a limit of 2 GB. So when my co-worker was depositing his package, Nginx accepted the deposit, and sent it to Tomcat. The trouble was that it wouldn’t send all data. When it finished with the 2GB part of the 8 GB size file it just stopped, while Tomcat was still waiting for the rest of the data. After a short while it timed out, and returned an HTTP code of 500 to Nginx. Turning off sendfile() fixes this, as Nginx can now send all the data, however this makes network I/O slower.

Drupal 8 entity access permission problems

Sometimes even when you think you’ve given all the correct permissions to a user role Drupal still denies access to a node. This happened to me this week at work unfortunately and I spent some time digging through Drupal 8’s codebase to track down where and why it does this. So in case someone else has this problem as well, these calls in order are the interesting code parts to check out.

AccessManager::check
AccessManager::performCheck
EntityAccessCheck::access
Node::access
ContentEntityBase::access
EntityAccessControlHandler::access

In my case from the bottom of this stack turned out to be one of the modules that denied access. EntityAccessControlHandler::access method calls ModuleHandler::invokeAll to query the modules if they will allow access. Let’s see that Drupal method in version 8.3.x:

public function invokeAll($hook, array $args = []) {
$return = [];
$implementations = $this->getImplementations($hook);
foreach ($implementations as $module) {
$function = $module . ‘_’ . $hook;
$result = call_user_func_array($function, $args);
if (isset($result) && is_array($result)) {
$return = NestedArray::mergeDeep($return, $result);
}
elseif (isset($result)) {
$return[] = $result;
}
}

return $return;
}

As we can see it calls a hook, and passes the arguments that go into the method. The hooks are the following typically:

entity_access
node_access

The arguments are the following:

[ $entity, $operation, $account ]

Operations in case of access can be the following:

create
read
update
delete

Using these information we can filter the calls and list which module(s) deny access and then check out what their problem is

Tomcat 7 servlet context parameters

In desktop applications we typically configure our application by reading a config file, and using the values found.

In a Java web application however there’s a better way, because there’s already a config file we can use, and this is the web.xml file, where we can place the so called context init parameters, that are loaded during the web application’s startup. In this article I’m going to show how this works.

There are 3 things we must do to make this all work

  • Create a so called ContextListener subclass that will listen to events such as context initialization (servlet startup).
  • Tell Tomcat we have such a listener class and that it should tell it about such events, by referencing our class in the web.xml file
  • Place the parameters and values in the web.xml file

This is how a ContextListener class looks like:

package something.something.darkside;

import javax.servlet.ServletContext;
import javax.servlet.ServletContextListener;
import javax.servlet.ServletContextEvent;

public class ContextListener implements ServletContextListener
{
@Override
public void contextInitialized( ServletContextEvent event )
{
final ServletContext context = event.getServletContext();
final String dbdriver = context.getInitParameter( “dbdriver” );
final String dburl = context.getInitParameter( “dburl” );
}

@Override
public void contextDestroyed( ServletContextEvent event )
{
}
}

As we can see the class has 2 methods that need to be implemented, the contextInitialized and contextDestroyed events. We can read the context parameters in the contextInitialized class and then do whatever we want with them.

Let’s now see the relevant parts of such a web.xml file:

<?xml version=”1.0″ encoding=”UTF-8″?>
<web-app xmlns=”http://java.sun.com/xml/ns/javaee&#8221;
xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance&#8221;
xsi:schemaLocation=”http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd&#8221;
version=”3.0″>
<context-param>
<param-name>dbdriver</param-name>
<param-value>com.mysql.jdbc.Driver</param-value>
</context-param>
<context-param>
<param-name>dburl</param-name>
<param-value>jdbc:mysql://localhost/database</param-value>
</context-param>
<listener>
<listener-class>something.something.darkside.ContextListener</listener-class>
</listener>

As we can see we have 2 parameters here dbdriver, and dburl, which are database connection configuration parameters in this case. Also we have the context event listener class references. If we restart Tomcat now when starting up the servlet it will tell the context listener about the startup and it can read the parameters.

That’s it. It’s this simple!

Authenticating users against Active Directory with PHP

A co-worker of mine asked how this can be done, since I’ve already had some experience querying AD with Java and PHP. It’s been some time ago so I had to think about a little but then I realized it’s easy. Just use LDAP bind!

Consider the following PHP code snippet:

$ldaphost = ‘ldap://dc.mycompany.local’;
$ldapport = 389;

$user = ‘DOMAIN\user’;
$password = ‘password’;

$ldapconn = @ldap_connect( $ldaphost, $ldapport );
if( $ldapconn )
{

ldap_set_option( $ldapconn, LDAP_OPT_PROTOCOL_VERSION, 3 );
ldap_set_option( $ldapconn, LDAP_OPT_REFERRALS, 0 );

$ok = @ldap_bind( $ldapconn, $user, $password );
if( $ok )
{
ldap_unbind( $ldapconn );
}
else
{
echo ‘Failed to bind to LDAP server:’ . “\n”;
echo ldap_error( $ldapconn ) . “\n”;
}
}
else
{
echo ‘Failed to connect to LDAP server’ . “\n”;
}

It is important to prefix the user’s name with the domain otherwise it won’t accept it and obviously in real life you’d want to use LDAPS and not send the password over the wire in clear text, but the basic idea is the same.

Remote debugging Tomcat7 servlets with Netbeans

At work we still use Tomcat 7 in production and I needed to set up debugging for various development systems. This article shows how to enable Tomcat 7 remote debugging

Enabling Tomcat 7 remote debugging via JDWP

Linux

I use Ubuntu 16.04 LTE so I’ll use that in the example, but other distros will not be that much different, except for the path and (re)starting the service of course.

  • Edit or create the file /usr/share/tomcat7/bin/setenv.sh and put in the following content:

    export JAVA_OPTS=”-Xdebug \
    -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n”

    Note: Obviously if the file already exists and it already has some content, then just add the parameters instead of adding the entire line.

  • Restart Tomcat

    sudo service tomcat7 restart

Windows

  • Go to the Tomcat binary directory, which is by default

    c:\Program Files\Apache Software Foundation\Tomcat 7.0\bin

  • Start the program Tomcat7w.exe
  • Switch to the java tab and add the following lines to the Java options textbox:

    -Xdebug
    -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n

    Note: It is important that each of the parameters should be added on separate lines, and that lines should have no whitespaces in the end!

  • Restarts Tomcat 7

    net stop tomcat7
    net start tomcat7

Attaching Netbeans debugger to Tomcat 7

Now that we have Tomcat running with the remote debugging on we can attach Netbeans to debug.

  • Click debug – attach debugger, a dialog box will appear
  • Select Java Debugger (JPDA) as the Debugger
  • Select SocketAttach as Connector
  • Fill in host / IP address to the host field
  • Fill in port to the Port field, in this example the port is 8787, but obviously it can be any non-taken port
  • Click OK
  • If everything went OK the debugging tab should show up showing the running threads

…and that’s it! Happy bug hunting!

Achievements of modern technology

It is often said that one picture can tell a thousand words. This is a picture of the tablet I bought in 2015 browsing the boot floppy I created in 1998. Yes that was 18 years ago.

img_20161007_185650

New linter integration plugins for KDevelop

Hi there!
I’ve just moved some linter integration plugins to the KDE infrastructure (scratch repos), therefore making them generally available.
They are fairly simple plugins, all 3 of them are alike in that they just run an external tool, and bring the results (the issues found) into KDevelop’s GUI. The found issues then will be in the problems toolview, in their own separate tab. The tools can check either a single file or all files in the project. You can see the workflow and configuration options in the videos included. There are also user’s manuals and tech manuals in the docs directories of each repo.

kdev-clangcheck
This plugin integrates Clang’s static code analysis feature, providing C/C++ static code analysis.

Git repository:
git://anongit.kde.org/scratch/laszlok/kdev-clangcheck.git

kdev-pylint
This plugin integrates a linter called Pylint, and as the name suggests it’s a Python code analyzer.

Git repository:
git://anongit.kde.org/scratch/laszlok/kdev-pylint.git

kdev-jshint
This plugin integrates a linter called JSHint, and as the name suggests it’s a Javascript code analyzer.

Git repository:
git://anongit.kde.org/scratch/laszlok/kdev-jshint.git

New GUI for kdev-cppcheck

Hi there!

As you all probably know CppCheck is static code analyzer tool for C and C++. KDevelop has a plugin that provides a front-end for it, and the plugin is called kdev-cppcheck.

The good news is I’ve updated it’s GUI and now it uses the KDevelop Problem Checker Framework.

In the past it used to have it’s own toolview, where it showed issues in different formats (flat issue list, grouped by files, grouped by issue severity), based on the settings set in a KCM module.

Here’s a screenshot showing and example of this

20150722_000003008

What I’ve done is break up that KCM module, and create a per project settings window, and a general global settings window. The global settings window allows you to set the location of the cppcheck tool

20150722_000003009

The per project settings window allows one to set the rest: parameters, and what should be checked

20150722_000003010

Also the results area now shown in the problems toolview, just like problems found by the background parser, in it’s own tab.

20150722_000003011

Here’s a video showing the workflow

New GUI for kdev-krazy2

Hi there!

First of all let me introduce some concepts for readers who are unfamilair with them.

What is krayz2?

Krazy2 is a set of code tests (basically static code analysis) for KDE developers.

What is kdev-krazy2?

kdev-krazy is a plugin for KDevelop, that provides a frontend for Krazy2, so it can be run directly from KDevelop. The resulting issues also show up in KDevelop.

What’s changed?

I’ve given some love to this plugin lately: First I ported it to KF5 so it can run in the latest KDevelop. Now I’ve changed it’s GUI so it now uses the new KDevelop Problem Checker Framework.

Up until now the plugin had it’s own toolview. That’s where settings could be changed, analysis started, and that’s where the issues showed up. Let’s see some screenshots!

The first one shows the main KDevelop window, with the plugin loaded, showing the krazy2 toolview docked in the bottom (fairly large picture, feel free to click).

20150720_000002991

Clicking either the “Select paths” or “Select checkers” buttons shows settings dialogs, not surprisingly you can select paths and chekers in them. The next 2 screenshots shows those.

20150720_000002992

20150720_000002993

Finally the result of the analysis is shown in the toolview

20150720_000002994

All this was in the past. Now the settings can be changed in the per project settings window

20150720_000002995

20150720_000002996

The analysis can be started from the Run menu.

20150720_000002997

The results show up in the problems toolview, the same way that problems detected by the background parser, in a separate tab

20150720_000002998

Here’s a video showing how it all works