PlugIn/Compatibility between Payara and Hotswap Agent #715

angerhard · 2016-03-22T12:11:35Z

Using Hotswap Agent (open source alternative to JRebel) with Payara for hot code swapping would be very useful and a true benefit to develop jee with it.

OndroMih · 2016-03-22T13:11:30Z

Would you be more specific? What are the issues when you try to use Hotswap Agent with Payara server?

Hotswap Agent requires to modify your JDK with a patch, and then add a java agent to the application server.
Therefore you need to update Payara server configuration (works also with Glassfish):

if you modify a separate JDK, you may point Payara server to this JDK using AS_JAVA property in PAYARA_INSTALL/glassfish/config/asenv (more details here )
to add the java agent, you need to specify correct JVM option
-- in admin console it is in server-config, "JVM options" tab

If above steps don't work, please post more details about your issues.

angerhard · 2016-03-23T22:17:59Z

Hello OndrejM,

thank you for your answer.

I tried to install the HotSwapAgent like you explaned. But I get the error:

[2016-03-23T22:18:17.617+0100] [Payara 4.1] [WARNING] [] [org.glassfish.jersey.internal.Errors] [tid: _ThreadID=78 _ThreadName=admin-listener(2)] [timeMillis: 1458767897617] [levelValue: 900] [[
  The following warnings have been detected: WARNING: Unknown HK2 failure detected:
MultiException stack 1 of 1
java.lang.NoClassDefFoundError: org/hotswap/agent/config/PluginManager

so I added the HotSwapAgent.jar into the domain1/lib/ext-Path and added some packages to the osgi.properties (like describe here: click )

The payara starts as normal and the server.log shows some started plugins. But when I try to change a session bean, Eclipse throws the standard error the code could not be swapped.

I attached the log file and some screenshots. I hope you can help me :-)

Many thanks in advance,

Andreas

server.log:

[2016-03-23T22:59:35.979+0100] [] [INFO] [NCLS-GFLAUNCHER-00005] [javax.enterprise.launcher] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1458770375979] [levelValue: 800] [[
  JVM invocation command line:
C:\opt\payara41hs\jdk1.8.0_74\bin\java.exe
-cp
C:/opt/payara41hs/glassfish/modules/glassfish.jar
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9009
-XX:+UnlockDiagnosticVMOptions
-XX:NewRatio=2
-XXaltjvm=dcevm
-Xmx512m
-javaagent:C:/opt/payara41hs/glassfish/lib/monitor/flashlight-agent.jar
-client
-javaagent:c:\opt\hotswap-agent.jar
-Djavax.xml.accessExternalSchema=all
-Djavax.net.ssl.trustStore=C:\opt\payara41hs\glassfish\domains\domain1/config/cacerts.jks
-Djdk.tls.rejectClientInitiatedRenegotiation=true
-Djdk.corba.allowOutputStreamSubclass=true
-Dfelix.fileinstall.dir=C:\opt\payara41hs\glassfish/modules/autostart/
-Dorg.glassfish.additionalOSGiBundlesToStart=org.apache.felix.shell,org.apache.felix.gogo.runtime,org.apache.felix.gogo.shell,org.apache.felix.gogo.command,org.apache.felix.shell.remote,org.apache.felix.fileinstall
-Dcom.sun.aas.installRoot=C:\opt\payara41hs\glassfish
-Dfelix.fileinstall.poll=5000
-Djava.endorsed.dirs=C:\opt\payara41hs\glassfish/modules/endorsed;C:\opt\payara41hs\glassfish/lib/endorsed
-Djava.security.policy=C:\opt\payara41hs\glassfish\domains\domain1/config/server.policy
-Dosgi.shell.telnet.maxconn=1
-Dfelix.fileinstall.bundles.startTransient=true
-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
-Dfelix.fileinstall.log.level=2
-Djavax.net.ssl.keyStore=C:\opt\payara41hs\glassfish\domains\domain1/config/keystore.jks
-Djava.security.auth.login.config=C:\opt\payara41hs\glassfish\domains\domain1/config/login.conf
-Dfelix.fileinstall.disableConfigSave=false
-Dfelix.fileinstall.bundles.new.start=true
-Dcom.sun.aas.instanceRoot=C:\opt\payara41hs\glassfish\domains\domain1
-Dosgi.shell.telnet.port=6666
-Dgosh.args=--nointeractive
-Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as
-Dorg.jboss.weld.serialization.beanIdentifierIndexOptimization=false
-Dosgi.shell.telnet.ip=127.0.0.1
-DANTLR_USE_DIRECT_CLASS_LOADING=true
-Djava.awt.headless=true
-Dcom.ctc.wstx.returnNullForDefaultNamespace=true
-Djava.ext.dirs=C:\opt\payara41hs\jdk1.8.0_74/lib/ext;C:\opt\payara41hs\jdk1.8.0_74/jre/lib/ext;C:\opt\payara41hs\glassfish\domains\domain1/lib/ext
-Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
-Djava.library.path=C:/opt/payara41hs/glassfish/lib;C:/opt/payara41hs/jdk1.8.0_74/jre/bin;C:/Windows/Sun/Java/bin;C:/Windows/System32;C:/Windows;C:/oraclexe/app/oracle/product/11.2.0/server/bin;C:/Python34;C:/Python34/Scripts;C:/ProgramData/Oracle/Java/javapath;C:/Windows/System32/wbem;C:/Windows/System32/WindowsPowerShell/v1.0;C:/Program Files/TortoiseSVN/bin;C:/opt/apache-maven-3.3.3/bin;C:/opt/payara41hs/jdk1.8.0_74/bin;C:/Program Files (x86)/WinSCP;C:/Program Files/OpenVPN/bin;C:/Program Files/Git/cmd;C:/Program Files/Git/mingw64/bin;C:/Program Files/Git/usr/bin;C:/Program Files (x86)/Skype/Phone;C:/Program Files/nodejs;C:/Users/AGerhard/AppData/Roaming/npm;C:/Development/android-sdk/tools;C:/opt/payara41hs/glassfish/domains/domain1/config
com.sun.enterprise.glassfish.bootstrap.ASMain
-upgrade
false
-domaindir
C:/opt/payara41hs/glassfish/domains/domain1
-read-stdin
true
-asadmin-args
--host,,,localhost,,,--port,,,4848,,,--secure=false,,,--terse=false,,,--echo=false,,,--interactive=true,,,start-domain,,,--verbose=false,,,--watchdog=false,,,--debug=false,,,--domaindir,,,C:\opt\payara41hs\glassfish\domains,,,domain1
-domainname
domain1
-instancename
server
-type
DAS
-verbose
false
-asadmin-classpath
C:/opt/payara41hs/glassfish/lib/client/appserver-cli.jar
-debug
false
-asadmin-classname
com.sun.enterprise.admin.cli.AdminMain]]

[2016-03-23T22:59:40.818+0100] [Payara 4.1] [INFO] [NCLS-LOGGING-00009] [javax.enterprise.logging] [tid: _ThreadID=21 _ThreadName=RunLevelControllerThread-1458770380664] [timeMillis: 1458770380818] [levelValue: 800] [[
  Running Payara Version: Payara Server  4.1.1.161.1 #badassfish (build 63)]]

[2016-03-23T22:59:40.821+0100] [Payara 4.1] [INFO] [NCLS-LOGGING-00010] [javax.enterprise.logging] [tid: _ThreadID=21 _ThreadName=RunLevelControllerThread-1458770380664] [timeMillis: 1458770380821] [levelValue: 800] [[
  Server log file is using Formatter class: com.sun.enterprise.server.logging.ODLLogFormatter]]

[2016-03-23T22:59:41.029+0100] [Payara 4.1] [INFO] [] [org.glassfish.ha.store.spi.BackingStoreFactoryRegistry] [tid: _ThreadID=23 _ThreadName=RunLevelControllerThread-1458770380678] [timeMillis: 1458770381029] [levelValue: 800] [[
  Registered org.glassfish.ha.store.adapter.cache.ShoalBackingStoreProxy for persistence-type = replicated in BackingStoreFactoryRegistry]]

[2016-03-23T22:59:41.226+0100] [Payara 4.1] [INFO] [] [org.glassfish.ha.store.spi.BackingStoreFactoryRegistry] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770381226] [levelValue: 800] [[
  Registered fish.payara.ha.hazelcast.store.HazelcastBackingStoreFactoryProxy for persistence-type = hazelcast in BackingStoreFactoryRegistry]]

[2016-03-23T22:59:41.227+0100] [Payara 4.1] [INFO] [] [fish.payara.ha.hazelcast.store.HazelcastBackingStoreFactory] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770381227] [levelValue: 800] [[
  Registered Hazelcast BackingStoreFactory with persistence-type = hazelcast]]

[2016-03-23T22:59:41.352+0100] [Payara 4.1] [INFO] [SEC-SVCS-00100] [javax.enterprise.security.services] [tid: _ThreadID=23 _ThreadName=RunLevelControllerThread-1458770380678] [timeMillis: 1458770381352] [levelValue: 800] [[
  Authorization Service has successfully initialized.]]

[2016-03-23T22:59:41.405+0100] [Payara 4.1] [INFO] [NCLS-SECURITY-01115] [javax.enterprise.system.core.security] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770381405] [levelValue: 800] [[
  Realm [admin-realm] of classtype [com.sun.enterprise.security.auth.realm.file.FileRealm] successfully created.]]

[2016-03-23T22:59:41.406+0100] [Payara 4.1] [INFO] [NCLS-SECURITY-01115] [javax.enterprise.system.core.security] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770381406] [levelValue: 800] [[
  Realm [file] of classtype [com.sun.enterprise.security.auth.realm.file.FileRealm] successfully created.]]

[2016-03-23T22:59:41.414+0100] [Payara 4.1] [INFO] [NCLS-SECURITY-01115] [javax.enterprise.system.core.security] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770381414] [levelValue: 800] [[
  Realm [certificate] of classtype [com.sun.enterprise.security.auth.realm.certificate.CertificateRealm] successfully created.]]

[2016-03-23T22:59:41.567+0100] [Payara 4.1] [INFO] [NCLS-CORE-00087] [javax.enterprise.system.core] [tid: _ThreadID=22 _ThreadName=RunLevelControllerThread-1458770380678] [timeMillis: 1458770381567] [levelValue: 800] [[
  Grizzly Framework 2.3.23 started in: 89ms - bound to [/0.0.0.0:8080]]]

[2016-03-23T22:59:41.592+0100] [Payara 4.1] [INFO] [NCLS-CORE-00087] [javax.enterprise.system.core] [tid: _ThreadID=22 _ThreadName=RunLevelControllerThread-1458770380678] [timeMillis: 1458770381592] [levelValue: 800] [[
  Grizzly Framework 2.3.23 started in: 2ms - bound to [/0.0.0.0:8181]]]

[2016-03-23T22:59:41.601+0100] [Payara 4.1] [INFO] [NCLS-CORE-00087] [javax.enterprise.system.core] [tid: _ThreadID=22 _ThreadName=RunLevelControllerThread-1458770380678] [timeMillis: 1458770381601] [levelValue: 800] [[
  Grizzly Framework 2.3.23 started in: 2ms - bound to [/0.0.0.0:4848]]]

[2016-03-23T22:59:41.649+0100] [Payara 4.1] [INFO] [NCLS-CORE-00087] [javax.enterprise.system.core] [tid: _ThreadID=21 _ThreadName=RunLevelControllerThread-1458770380664] [timeMillis: 1458770381649] [levelValue: 800] [[
  Grizzly Framework 2.3.23 started in: 1ms - bound to [/0.0.0.0:3700]]]

[2016-03-23T22:59:41.944+0100] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770381944] [levelValue: 1000] [[
  Cannot find the resource bundle for the name com.sun.logging.enterprise.resource.corba for class org.glassfish.enterprise.iiop.impl.GlassFishORBManager using org.glassfish.main.orb.iiop [222]]]

[2016-03-23T22:59:42.260+0100] [Payara 4.1] [INFO] [NCLS-SECURITY-01002] [javax.enterprise.system.core.security] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770382260] [levelValue: 800] [[
  Java security manager is disabled.]]

[2016-03-23T22:59:42.261+0100] [Payara 4.1] [INFO] [NCLS-SECURITY-01010] [javax.enterprise.system.core.security] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770382261] [levelValue: 800] [[
  Entering Security Startup Service.]]

[2016-03-23T22:59:42.265+0100] [Payara 4.1] [INFO] [NCLS-SECURITY-01143] [javax.enterprise.system.core.security] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770382265] [levelValue: 800] [[
  Loading policy provider com.sun.enterprise.security.provider.PolicyWrapper.]]

[2016-03-23T22:59:42.327+0100] [Payara 4.1] [INFO] [NCLS-SECURITY-01011] [javax.enterprise.system.core.security] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770382327] [levelValue: 800] [[
  Security Service(s) started successfully.]]

[2016-03-23T22:59:42.634+0100] [Payara 4.1] [INFO] [AS-WSMETROGLUE-10010] [javax.enterprise.webservices.metroglue] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770382634] [levelValue: 800] [[
  Web service endpoint deployment events listener registered successfully.]]

[2016-03-23T22:59:43.680+0100] [Payara 4.1] [INFO] [AS-WSMETROGLUE-10001] [javax.enterprise.webservices.metroglue] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770383680] [levelValue: 800] [[
  Loading WS-TX Services. Please wait.]]

[2016-03-23T22:59:44.691+0100] [Payara 4.1] [INFO] [AS-WEB-GLUE-00198] [javax.enterprise.web] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770384691] [levelValue: 800] [[
  Created HTTP listener http-listener-1 on host/port 0.0.0.0:8080]]

[2016-03-23T22:59:44.700+0100] [Payara 4.1] [INFO] [AS-WEB-GLUE-00198] [javax.enterprise.web] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770384700] [levelValue: 800] [[
  Created HTTP listener http-listener-2 on host/port 0.0.0.0:8181]]

[2016-03-23T22:59:44.702+0100] [Payara 4.1] [INFO] [AS-WEB-GLUE-00198] [javax.enterprise.web] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770384702] [levelValue: 800] [[
  Created HTTP listener admin-listener on host/port 0.0.0.0:4848]]

[2016-03-23T22:59:44.734+0100] [Payara 4.1] [INFO] [AS-WEB-GLUE-00200] [javax.enterprise.web] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770384734] [levelValue: 800] [[
  Created virtual server server]]

[2016-03-23T22:59:44.737+0100] [Payara 4.1] [INFO] [AS-WEB-GLUE-00200] [javax.enterprise.web] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770384737] [levelValue: 800] [[
  Created virtual server __asadmin]]

[2016-03-23T22:59:45.037+0100] [Payara 4.1] [INFO] [AS-WEB-CORE-00306] [javax.enterprise.web.core] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770385037] [levelValue: 800] [[
  Setting JAAS app name glassfish-web]]

[2016-03-23T22:59:45.039+0100] [Payara 4.1] [INFO] [AS-WEB-GLUE-00201] [javax.enterprise.web] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770385039] [levelValue: 800] [[
  Virtual server server loaded default web module ]]

[2016-03-23T22:59:45.350+0100] [Payara 4.1] [INFO] [] [] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770385350] [levelValue: 800] [[
  HOTSWAP AGENT: 22:59:45.350 INFO (org.hotswap.agent.config.PluginRegistry) - Plugin 'org.hotswap.agent.plugin.tomcat.TomcatPlugin' initialized in ClassLoader 'WebappClassLoader (delegate=true; repositories=)'.]]

[2016-03-23T22:59:45.350+0100] [Payara 4.1] [INFO] [] [] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770385350] [levelValue: 800] [[
  HOTSWAP AGENT: 22:59:45.350 INFO (org.hotswap.agent.plugin.tomcat.TomcatPlugin) - Tomcat plugin initialized - Tomcat version 'unknown']]

[2016-03-23T22:59:46.633+0100] [Payara 4.1] [INFO] [AS-WSJSR109IMPL-00018] [javax.enterprise.webservices] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770386633] [levelValue: 800] [[
  Webservice Endpoint deployed RegistrationRequesterPortTypePortImpl
 listening at address at http://VIVEC:8080/__wstx-services/RegistrationRequesterPortType11.]]

[2016-03-23T22:59:46.635+0100] [Payara 4.1] [INFO] [AS-WSJSR109IMPL-00018] [javax.enterprise.webservices] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770386635] [levelValue: 800] [[
  Webservice Endpoint deployed RegistrationPortTypeRPCPortImpl
 listening at address at http://VIVEC:8080/__wstx-services/RegistrationPortTypeRPC11.]]

[2016-03-23T22:59:46.636+0100] [Payara 4.1] [INFO] [AS-WSJSR109IMPL-00018] [javax.enterprise.webservices] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770386636] [levelValue: 800] [[
  Webservice Endpoint deployed CoordinatorPortTypeImpl
 listening at address at http://VIVEC:8080/__wstx-services/CoordinatorPortType.]]

[2016-03-23T22:59:46.637+0100] [Payara 4.1] [INFO] [AS-WSJSR109IMPL-00018] [javax.enterprise.webservices] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770386637] [levelValue: 800] [[
  Webservice Endpoint deployed ParticipantPortTypeImpl
 listening at address at http://VIVEC:8080/__wstx-services/ParticipantPortType.]]

[2016-03-23T22:59:46.638+0100] [Payara 4.1] [INFO] [AS-WSJSR109IMPL-00018] [javax.enterprise.webservices] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770386638] [levelValue: 800] [[
  Webservice Endpoint deployed ParticipantPortTypePortImpl
 listening at address at http://VIVEC:8080/__wstx-services/ParticipantPortType11.]]

[2016-03-23T22:59:46.639+0100] [Payara 4.1] [INFO] [AS-WSJSR109IMPL-00018] [javax.enterprise.webservices] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770386639] [levelValue: 800] [[
  Webservice Endpoint deployed RegistrationRequesterPortTypeImpl
 listening at address at http://VIVEC:8080/__wstx-services/RegistrationRequesterPortType.]]

[2016-03-23T22:59:46.640+0100] [Payara 4.1] [INFO] [AS-WSJSR109IMPL-00018] [javax.enterprise.webservices] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770386640] [levelValue: 800] [[
  Webservice Endpoint deployed RegistrationPortTypeImpl
 listening at address at http://VIVEC:8080/__wstx-services/RegistrationPortTypeRPC.]]

[2016-03-23T22:59:46.641+0100] [Payara 4.1] [INFO] [AS-WSJSR109IMPL-00018] [javax.enterprise.webservices] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770386641] [levelValue: 800] [[
  Webservice Endpoint deployed CoordinatorPortTypePortImpl
 listening at address at http://VIVEC:8080/__wstx-services/CoordinatorPortType11.]]

[2016-03-23T22:59:47.292+0100] [Payara 4.1] [INFO] [] [] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770387292] [levelValue: 800] [[
  HOTSWAP AGENT: 22:59:47.292 INFO (org.hotswap.agent.config.PluginRegistry) - Plugin 'org.hotswap.agent.plugin.tomcat.TomcatPlugin' initialized in ClassLoader 'WebappClassLoader (delegate=true; repositories=WEB-INF/classes/)'.]]

[2016-03-23T22:59:47.292+0100] [Payara 4.1] [INFO] [] [] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770387292] [levelValue: 800] [[
  HOTSWAP AGENT: 22:59:47.292 INFO (org.hotswap.agent.plugin.tomcat.TomcatPlugin) - Tomcat plugin initialized - Tomcat version 'unknown']]

[2016-03-23T22:59:52.407+0100] [Payara 4.1] [INFO] [AS-WEB-GLUE-00172] [javax.enterprise.web] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770392407] [levelValue: 800] [[
  Loading application [wstx-services] at [/__wstx-services]]]

[2016-03-23T22:59:52.423+0100] [Payara 4.1] [INFO] [AS-WSMETROGLUE-10002] [javax.enterprise.webservices.metroglue] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770392423] [levelValue: 800] [[
  WS-TX Services successfully started.]]

[2016-03-23T22:59:52.796+0100] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770392796] [levelValue: 1000] [[
  Cannot find the resource bundle for the name com.sun.logging.enterprise.system.tools.deployment for class com.sun.ejb.codegen.ServiceInterfaceGenerator using org.glassfish.main.ejb.ejb-container [73]]]

[2016-03-23T22:59:53.307+0100] [Payara 4.1] [WARNING] [] [com.sun.xml.ws.transport.http.servlet.ServletAdapter] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770393307] [levelValue: 900] [[
  Container org.glassfish.webservices.JAXWSContainer@1f009de doesn't support class com.sun.xml.ws.api.server.Module]]

[2016-03-23T22:59:53.455+0100] [Payara 4.1] [INFO] [AS-EJB-00054] [javax.enterprise.ejb.container] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770393455] [levelValue: 800] [[
  Portable JNDI names for EJB DemoBean: [java:global/serv/DemoBean, java:global/serv/DemoBean!de.servicepack7.gh.server.DemoBean]]]

[2016-03-23T22:59:53.614+0100] [Payara 4.1] [INFO] [] [org.jboss.weld.Version] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770393614] [levelValue: 800] [[
  WELD-000900: 2.2.16 (Final)]]

[2016-03-23T22:59:54.488+0100] [Payara 4.1] [INFO] [] [org.hibernate.validator.internal.util.Version] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770394488] [levelValue: 800] [[
  HV000001: Hibernate Validator 5.1.2.Final]]

[2016-03-23T22:59:54.751+0100] [Payara 4.1] [INFO] [] [] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770394751] [levelValue: 800] [[
  HOTSWAP AGENT: 22:59:54.751 INFO (org.hotswap.agent.config.PluginRegistry) - Plugin 'org.hotswap.agent.plugin.hibernate.HibernatePlugin' initialized in ClassLoader 'org.glassfish.hk2.external.bean-validator [23]'.]]

[2016-03-23T22:59:55.827+0100] [Payara 4.1] [INFO] [] [org.jboss.weld.Event] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770395827] [levelValue: 800] [[
  WELD-000411: Observer method [BackedAnnotatedMethod] private org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider.processAnnotatedType(@Observes ProcessAnnotatedType<Object>) receives events for all annotated types. Consider restricting events using @WithAnnotations or a generic type with bounds.]]

[2016-03-23T22:59:57.718+0100] [Payara 4.1] [INFO] [AS-WSJSR109IMPL-00019] [javax.enterprise.webservices] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770397718] [levelValue: 800] [[
  EJB Endpoint deployed serv
  listening at address at http://VIVEC:8080/DemoBeanService/DemoBean]]

[2016-03-23T22:59:57.720+0100] [Payara 4.1] [INFO] [NCLS-CORE-00022] [javax.enterprise.system.core] [tid: _ThreadID=20 _ThreadName=RunLevelControllerThread-1458770380641] [timeMillis: 1458770397720] [levelValue: 800] [[
  Loading application serv done in 16.264 ms]]

[2016-03-23T22:59:57.722+0100] [Payara 4.1] [INFO] [NCLS-CORE-00017] [javax.enterprise.system.core] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1458770397722] [levelValue: 800] [[
  Payara Server  4.1.1.161.1 #badassfish (63) startup time : Felix (3.089ms), startup services(17.089ms), total(20.178ms)]]

[2016-03-23T22:59:57.724+0100] [Payara 4.1] [INFO] [] [fish.payara.nucleus.util.CleanupPostBoot] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1458770397724] [levelValue: 800] [[
  Cleaning JarFileFactory Cache to prevent jar FD leaks]]

[2016-03-23T22:59:57.860+0100] [Payara 4.1] [INFO] [NCLS-CORE-00087] [javax.enterprise.system.core] [tid: _ThreadID=21 _ThreadName=RunLevelControllerThread-1458770380664] [timeMillis: 1458770397860] [levelValue: 800] [[
  Grizzly Framework 2.3.23 started in: 3ms - bound to [/0.0.0.0:7676]]]

[2016-03-23T22:59:58.314+0100] [Payara 4.1] [INFO] [NCLS-JMX-00005] [javax.enterprise.system.jmx] [tid: _ThreadID=68 _ThreadName=Thread-20] [timeMillis: 1458770398314] [levelValue: 800] [[
  JMXStartupService has started JMXConnector on JMXService URL service:jmx:rmi://VIVEC:8686/jndi/rmi://VIVEC:8686/jmxrmi]]

[2016-03-23T23:00:01.945+0100] [Payara 4.1] [INFO] [NCLS-REST-00001] [javax.enterprise.admin.rest] [tid: _ThreadID=46 _ThreadName=admin-listener(3)] [timeMillis: 1458770401945] [levelValue: 800] [[
  Listening to REST requests at context: /management/domain.]]

[2016-03-23T23:00:02.182+0100] [Payara 4.1] [INFO] [NCLS-BOOTSTRAP-00027] [javax.enterprise.bootstrap] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1458770402182] [levelValue: 800] [[
  Registered com.sun.enterprise.glassfish.bootstrap.osgi.EmbeddedOSGiGlassFishImpl@3d7057 as OSGi service registration: org.apache.felix.framework.ServiceRegistrationImpl@1bb1a4.]]

[2016-03-23T23:00:02.307+0100] [Payara 4.1] [INFO] [] [] [tid: _ThreadID=18 _ThreadName=FelixStartLevel] [timeMillis: 1458770402307] [levelValue: 800] [[
  C:\opt\payara41hs\glassfish\domains\domain1\autodeploy\bundles does not exist, please create it.]]

[2016-03-23T23:00:03.188+0100] [Payara 4.1] [INFO] [] [] [tid: _ThreadID=87 _ThreadName=Thread-22] [timeMillis: 1458770403188] [levelValue: 800] [[
  HOTSWAP AGENT: 23:0:3.188 INFO (org.hotswap.agent.config.PluginRegistry) - Plugin 'org.hotswap.agent.plugin.tomcat.TomcatPlugin' initialized in ClassLoader 'WebappClassLoader (delegate=true; repositories=WEB-INF/classes/)'.]]

[2016-03-23T23:00:03.189+0100] [Payara 4.1] [INFO] [] [] [tid: _ThreadID=87 _ThreadName=Thread-22] [timeMillis: 1458770403189] [levelValue: 800] [[
  HOTSWAP AGENT: 23:0:3.189 INFO (org.hotswap.agent.plugin.tomcat.TomcatPlugin) - Tomcat plugin initialized - Tomcat version 'unknown']]

[2016-03-23T23:00:05.947+0100] [Payara 4.1] [INFO] [jsf.config.listener.version] [javax.enterprise.resource.webcontainer.jsf.config] [tid: _ThreadID=87 _ThreadName=Thread-22] [timeMillis: 1458770405947] [levelValue: 800] [[
  Mojarra 2.2.12 ( 20150720-0848 https://svn.java.net/svn/mojarra~svn/tags/2.2.12@14885) für Kontext '' wird initialisiert.]]

[2016-03-23T23:00:09.664+0100] [Payara 4.1] [INFO] [] [] [tid: _ThreadID=87 _ThreadName=Thread-22] [timeMillis: 1458770409664] [levelValue: 800] [[
  HOTSWAP AGENT: 23:0:9.664 INFO (org.hotswap.agent.config.PluginRegistry) - Plugin 'org.hotswap.agent.plugin.jsf.JsfPlugin' initialized in ClassLoader 'org.glassfish.javax.faces [144]'.]]

[2016-03-23T23:00:09.829+0100] [Payara 4.1] [INFO] [] [] [tid: _ThreadID=87 _ThreadName=Thread-22] [timeMillis: 1458770409829] [levelValue: 800] [[
  HOTSWAP AGENT: 23:0:9.828 INFO (org.hotswap.agent.config.PluginRegistry) - Plugin 'org.hotswap.agent.plugin.elresolver.ELResolverPlugin' initialized in ClassLoader 'com.sun.el.javax.el [140]'.]]

[2016-03-23T23:00:10.086+0100] [Payara 4.1] [INFO] [AS-WEB-GLUE-00172] [javax.enterprise.web] [tid: _ThreadID=87 _ThreadName=Thread-22] [timeMillis: 1458770410086] [levelValue: 800] [[
  Loading application [__admingui] at [/]]]

[2016-03-23T23:00:10.088+0100] [Payara 4.1] [INFO] [NCLS-CORE-00022] [javax.enterprise.system.core] [tid: _ThreadID=87 _ThreadName=Thread-22] [timeMillis: 1458770410088] [levelValue: 800] [[
  Loading application __admingui done in 7.907 ms]]

OndroMih · 2016-03-29T16:41:56Z

@angerhard, could you try installing DCEVM using the other option - "Replace by DCEVM"? You should not use an option -XXaltjvm=dcevm then, only -javaagent...

I tried to setup DCEVM and HotSwap agent with Payara and Netbeans on Ubuntu Linux, with Java 8, and also received errors in Netbeans when I tried to rename a method - the same thing I would get if using just normal hot replace with Hotspot JVM from Oracle. I was using altjvm installation option.

After a while I tried to install DCEVM using option "Replace by DCEVM", and then hot replace worked sesamlessly when I changed method name.

So I suppose that Payara works with HotSwap agent, but the installation option as altjvm does not work. Possibly this is an issue with the Oracle JVM itself, as it should consider the -XXaltjvm option and load JVM from dcevm directory in JRE lib folder.

As DCEVM developers would possible know more, I created an issue for DCEVM:
dcevm/dcevm#97

OndroMih · 2016-03-29T19:37:50Z

@angerhard, some good news :) I found out that HotSwap doesn't work with -XXaltjvm option, when there is already -client or -server option in the parameters. Payara default domain already contains -client JVM option, so please try to remove it when adding -XXaltjvm=dcevm option.

angerhard · 2016-03-29T20:36:08Z

Hi OndrejM,

yes, I tried both ways, but don't get any of them running.

I tried following simple code (I tried to test something simple):

import java.util.logging.Logger;

import javax.ejb.Stateless;
import javax.jws.WebMethod;
import javax.jws.WebService;

@Stateless
@WebService
public class DemoBean {

    @WebMethod
    public void testSomething() {
        simple();
    }

    private void simple() {
        Logger.getAnonymousLogger().severe("xxx ");
        Thread.dumpStack();
    }
}

(and started the webservice over the installed Tester). Sometimes (but not deterministic) I received a warning in the server.log:

[2016-03-29T22:03:26.304+0200] [Payara 4.1] [WARNING] [NCLS-COMUTIL-00016] [javax.enterprise.system.util] [tid: _ThreadID=6 _ThreadName=JDWP Transport Listener: dt_socket] [timeMillis: 1459281806304] [levelValue: 900] [[
  Error: Request made to load class or resource [hotswap-agent.properties] on an ASURLClassLoader instance that has already been shutdown. [ASURLClassLoader.done() called ON ASURLClassLoader : 
urlSet = [URLEntry : file:/C:/opt/payara41hs/glassfish/domains/domain1/applications/serv/, URLEntry : file:/C:/opt/payara41hs/glassfish/domains/domain1/generated/ejb/serv/]
doneCalled = false 
 Parent -> org.glassfish.internal.api.DelegatingClassLoader@4fec430f

 AT Tue Mar 29 22:02:38 CEST 2016 
 BY :[java.lang.Thread.getStackTrace(Thread.java:1552), com.sun.enterprise.loader.ASURLClassLoader.done(ASURLClassLoader.java:217), com.sun.enterprise.loader.ASURLClassLoader.preDestroy(ASURLClassLoader.java:185), org.glassfish.deployment.common.DeploymentContextImpl.getClassLoader(DeploymentContextImpl.java:266), org.glassfish.deployment.common.DeploymentContextImpl.getClassLoader(DeploymentContextImpl.java:216), com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:485), com.sun.enterprise.v3.server.ApplicationLoaderService.processApplication(ApplicationLoaderService.java:406), com.sun.enterprise.v3.server.ApplicationLoaderService.postConstruct(ApplicationLoaderService.java:243), org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:326), org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:374), org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:471), org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:228), org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:85), org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2022), org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:114), org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:88), org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213), org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144), java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142), java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617), java.lang.Thread.run(Thread.java:745)]]]]

I think I am overssing something simple ...

C:\opt\payara41hs\jdk1.8.0_74\bin\java.exe
-cp
C:/opt/payara41hs/glassfish/modules/glassfish.jar
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9009
-XX:+UnlockDiagnosticVMOptions
-XX:NewRatio=2
-XXaltjvm=dcevm
-Xmx512m
-javaagent:C:/opt/payara41hs/glassfish/lib/monitor/flashlight-agent.jar
-javaagent:c:\opt\hotswap-agent.jar
-Djavax.xml.accessExternalSchema=all
-Djavax.net.ssl.trustStore=C:\opt\payara41hs\glassfish\domains\domain1/config/cacerts.jks
-Djdk.tls.rejectClientInitiatedRenegotiation=true
-Djdk.corba.allowOutputStreamSubclass=true
-Dfelix.fileinstall.dir=C:\opt\payara41hs\glassfish/modules/autostart/
-Dorg.glassfish.additionalOSGiBundlesToStart=org.apache.felix.shell,org.apache.felix.gogo.runtime,org.apache.felix.gogo.shell,org.apache.felix.gogo.command,org.apache.felix.shell.remote,org.apache.felix.fileinstall
-Dcom.sun.aas.installRoot=C:\opt\payara41hs\glassfish
-Dfelix.fileinstall.poll=5000
-Djava.security.policy=C:\opt\payara41hs\glassfish\domains\domain1/config/server.policy
-Djava.endorsed.dirs=C:\opt\payara41hs\glassfish/modules/endorsed;C:\opt\payara41hs\glassfish/lib/endorsed
-Dfelix.fileinstall.bundles.startTransient=true
-Dosgi.shell.telnet.maxconn=1
-Dfelix.fileinstall.log.level=2
-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
-Djavax.net.ssl.keyStore=C:\opt\payara41hs\glassfish\domains\domain1/config/keystore.jks
-Djava.security.auth.login.config=C:\opt\payara41hs\glassfish\domains\domain1/config/login.conf
-Dfelix.fileinstall.disableConfigSave=false
-Dfelix.fileinstall.bundles.new.start=true
-Dcom.sun.aas.instanceRoot=C:\opt\payara41hs\glassfish\domains\domain1
-Dosgi.shell.telnet.port=6666
-Dgosh.args=--nointeractive
-Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as
-Dorg.jboss.weld.serialization.beanIdentifierIndexOptimization=false
-Dosgi.shell.telnet.ip=127.0.0.1
-DANTLR_USE_DIRECT_CLASS_LOADING=true
-Dcom.ctc.wstx.returnNullForDefaultNamespace=true
-Djava.awt.headless=true
-Djava.ext.dirs=C:\opt\payara41hs\jdk1.8.0_74/lib/ext;C:\opt\payara41hs\jdk1.8.0_74/jre/lib/ext;C:\opt\payara41hs\glassfish\domains\domain1/lib/ext
-Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
-Djava.library.path=C:/opt/payara41hs/glassfish/lib;C:/ProgramData/Oracle/Java/javapath;C:/Windows/Sun/Java/bin;C:/Windows/System32;C:/Windows;C:/opt/payara41hs/%CommonProgramFiles%/Microsoft Shared/Windows Live;C:/oraclexe/app/oracle/product/11.2.0/server/bin;C:/Python34;C:/Python34/Scripts;C:/Windows/System32/wbem;C:/Windows/System32/WindowsPowerShell/v1.0;C:/Program Files/TortoiseSVN/bin;C:/opt/apache-maven-3.3.3/bin;C:/opt/payara41hs/jdk1.8.0_74/bin;C:/Program Files (x86)/WinSCP;C:/Program Files/OpenVPN/bin;C:/Program Files/Git/cmd;C:/Program Files/Git/mingw64/bin;C:/Program Files/Git/usr/bin;C:/Program Files (x86)/Skype/Phone;C:/Program Files/nodejs;C:/Program Files/Common Files/Microsoft Shared/Windows Live;C:/Users/AGerhard/AppData/Roaming/npm;C:/Development/android-sdk/tools;C:/opt/payara41hs
com.sun.enterprise.glassfish.bootstrap.ASMain

OndroMih · 2016-03-29T22:31:00Z

Are you sure you patched the following Java installation?

C:\opt\payara41hs\jdk1.8.0_74

Can you find this folder: C:\opt\payara41hs\jdk1.8.0_74\jre\lib\amd64\dcevm ?

angerhard · 2016-03-30T09:28:47Z

I havn't access on this machine by now, will check it in the evening. But I got hotswap-messages in the server.log, so I am very sure the jdk is patched (I have build up a simple installation for this test).
I will build up the same installation on my laptop (Ubuntu), too. Maybe some differences between win and linux? Can only guess now ...

OndroMih · 2016-03-30T15:32:59Z

You get hotswap messages with any JRE, even the unpatched one. The messages come from the java agent, which works to some extent even with an unpatched JRE (relaoding of framework specific files), but java code replacement is limited to what unpatched JRE offers.

It is not easy to find out from log whether you are running a patched JRE or unpatched, not even in jVisuaVM - only once you try code replacement with unsupported changes, then you see if it works or not.

angerhard · 2016-03-30T19:52:42Z

Ah ... I see.
OK, the JRE is patched, but the patcher put the altvm to C:\opt\payara41hs\jdk1.8.0_74\jre\bin\dcevm. I moved it to the path given by you with no further result. Da*ned, I think this case is easy to solve and I am really grateful for your help (and your patience).
I will build up the settings in ubuntu now, but will not get any results this evening. I will continue testing tomorrow.

OndroMih · 2016-03-31T14:58:41Z

I was using Linux, I checked on Windows now and the folder structure is different there. So it appears that the installer put it into the correct place in jre\bin\dcevm. The hint is that there is also folder called server, which is the defult JVM used.

OndroMih · 2016-03-31T16:23:40Z

I checked now on a Windows box and I was able to get hotswap working properly. Maybe the problem is with a particular type of change in the code, which is not handled even by the HotSwap DCEVM patch.

You are getting error "verifier detected internal inconsistency" - it might be caused by an inconsistent change, see the javadoc of the exception: http://ulibgcj.sourceforge.net/javadoc/java/lang/VerifyError.html
Doesn't say very much to me, but maybe it clicks when you think of what change you are doing.

To verify that hotswap patch is in place, I did plain rename of a method, and afterwards changed body of the method. The code was repaced and the app behaved differenty.

OndroMih · 2016-07-03T09:52:52Z

@angerhard, were you able to get Hotswap running with Payara? Were my instructions helpful?

OndroMih · 2016-08-17T06:24:00Z

I haven't found any issues using HotSwap Agent with Payara Server on Linux and Windows. However, the configuration to get it running can be a bit tricky.
We plan to write a blog post on blog.payara.fish to describe how to use HotSwap Agent with Payara Server.
I'm closing this issue now.

devauxbr · 2017-07-17T21:47:28Z

Hi, any update on a detailed blog post about how to use HotSwap Agent with Payara Server since this task was closed ? Couldn't find it with Google

Would be much appreciated ! :)

OndroMih · 2017-08-05T20:30:34Z

No blog is ready yet, but still in the plan.

With the latest version of HotSwap Agent, the following steps should be enough:

install the DCEVM JVM patch into the JVM used to start Payara Server, as an ALTJVM (see instructions in http://hotswapagent.org/ where to get the installer. You may need to run the installer as root / admin user)
copy the hotswap-agent.jar into domain/lib/ext directory
edit the JVM properties (e.g. via Admin console):
- delete the option -client
- add option -XXaltjvm=dcevm
- add option -javaagent:${com.sun.aas.instanceRoot}/lib/ext/hotswap-agent.jar (just copy this value, it references the domain directory via a variable)
run Payara Server in debug mode

In Netbeans:

in Project properties, disable Deploy on Save in the Run category
In Netbeans properties, go to Java Debugger options and check "Apply code changes after save"
run your application using "Debug project" from Netbeans

In Eclipse:

open Servers window and double click on the line with Payara Server, then check "Never publish automatically" in the section named Publishing
run application using "Debug on Server"

I haven't tried yet in IntelliJ, but it should also work while debugging an application and if application redeploy/automatic publishing is disabled.

LordFirlefanz · 2018-02-01T15:09:16Z

Hi there!

First: Tanks a lot for the plan/blog about hot-swapping! Just tried it and ... wow, very nice! Since a full deploy of our application takes up to 60 s this might be a blessing! :)

"Might" because one thing does not work and I'm not shure why.

Our application is an EAR-Application.
When I make a change in the code of the EJB-project, the corresponding class in the exploded gf-deploy-directory is going to be updated immediately and the new code works like a charm. Great!
But when I do the same on the WAR-side, on a managed bean e.g., it does not work. The classfile also is replaced with the fresh compiled one, but no change in behaviour. Seems like payara ist not recognizing the new classfile.
When I do a change in a xhtml e.g. it works in turn.

Any ideas would be great! What do I miss?

FISH-6066 (Enterprise) Invalid property 'default-web-xml' on instance start-up

OndroMih added the question label Mar 22, 2016

OndroMih added try to reproduce and removed question labels Mar 29, 2016

OndroMih self-assigned this Mar 29, 2016

OndroMih added reproducible and removed try to reproduce labels Mar 29, 2016

OndroMih added c:Question labels Apr 11, 2016

OndroMih added the requestor unresponsive (1) label Jul 3, 2016

OndroMih mentioned this issue Jul 3, 2016

is there a way to hot deploy of part of application? #886

Closed

OndroMih closed this as completed Aug 17, 2016

OndroMih removed the 1:Investigating label Aug 17, 2016

arieki pushed a commit to arieki/Payara that referenced this issue Dec 16, 2022

Merge pull request payara#715 from JamesHillyard/FISH-6066-Ent

c0765c3

FISH-6066 (Enterprise) Invalid property 'default-web-xml' on instance start-up

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PlugIn/Compatibility between Payara and Hotswap Agent #715

PlugIn/Compatibility between Payara and Hotswap Agent #715

angerhard commented Mar 22, 2016

OndroMih commented Mar 22, 2016

angerhard commented Mar 23, 2016

OndroMih commented Mar 29, 2016

OndroMih commented Mar 29, 2016

angerhard commented Mar 29, 2016

OndroMih commented Mar 29, 2016

angerhard commented Mar 30, 2016

OndroMih commented Mar 30, 2016

angerhard commented Mar 30, 2016

OndroMih commented Mar 31, 2016

OndroMih commented Mar 31, 2016

OndroMih commented Jul 3, 2016

OndroMih commented Aug 17, 2016

devauxbr commented Jul 17, 2017

OndroMih commented Aug 5, 2017

LordFirlefanz commented Feb 1, 2018

PlugIn/Compatibility between Payara and Hotswap Agent #715

PlugIn/Compatibility between Payara and Hotswap Agent #715

Comments

angerhard commented Mar 22, 2016

OndroMih commented Mar 22, 2016

angerhard commented Mar 23, 2016

OndroMih commented Mar 29, 2016

OndroMih commented Mar 29, 2016

angerhard commented Mar 29, 2016

OndroMih commented Mar 29, 2016

angerhard commented Mar 30, 2016

OndroMih commented Mar 30, 2016

angerhard commented Mar 30, 2016

OndroMih commented Mar 31, 2016

OndroMih commented Mar 31, 2016

OndroMih commented Jul 3, 2016

OndroMih commented Aug 17, 2016

devauxbr commented Jul 17, 2017

OndroMih commented Aug 5, 2017

LordFirlefanz commented Feb 1, 2018