Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.2
    • Fix Version/s: 2.3
    • Component/s: General
    • Labels:
      None

      Description

      LAMS becomes a large project even though it does not use such highly memory/CPU consuming technologies. There is a need to reduce the usage of resources to a reasonable level. Also, any tuning that will make LAMS services execute faster should also be taken into account.

        Activity

        marcin Marcin Cieslak created issue -
        Hide
        marcin Marcin Cieslak added a comment -
        Explanation of implemented changes:

        ================================================
        Spring
        1. Common application context
        All tools, admin, learning, monitoring etc. are Web applications. They might be modules of a single LAMS EAR, but for JBoss they are simply deployable WARs. It also means that they have separate application context. For each Web applications there is a root context, which is defined in our applicationContext.xml files. This context is common for all servlets defined in a Web application. Each servlet has also its own private context, but we weren't using this feature much. We were just using the shared context.

        Each of our tools (also learning, monitoring etc.) was importing some of application context config files from Central, Common etc. containing common beans like user management service. Also, each tool was adding some of its tool-specific beans. Event though those common beans were supposed to be shared and were declared as singletons, each of the tools' applications contexts were isolated from each other. This means each tool created its OWN SET OF SERVICES. Each common service and DAO had a copy in each tool.

        Additionally, our core services were importing all of our tools' application context files. Imports were added during tool deployment by appending lines into XML files in central, learning and monitoring.

        Spring has a class named ContextSingletonBeanFactoryLocator. This class allows searching for special configuration files (default: beanRefContext.xml) placed in classpath. These files contain bean definitions. Each bean is an application context. Each bean points to applicationContext.xml files that create this context. Beans can use inheritance. Locator gathers all special files it can find, glues them together and instantiates the beans (application contexts and beans within). After that these application context can be used as parents for Web application's root contexts. This means that each Web application (like one of our tools) can obtain such common application context and make it a parent of its root context, thus gaining access to all the beans. Those parent application context are instantiated only once, no matter which Web application calls for them (this is the "Singleton" word in class' name).

        2 common application contexts have been created:

        "context.tool" - for tools' use. As tools don't use all the core services, this context defines only the required ones plus all of the private tools' contexts. Tools' contexts are searched for in the classpath automatically, using ant-style parts. This mechanism locates and adds all the tools' applicationContext.xml files.

        "context.central" - inherits from above. It adds all the remaining services like monitoring. It is used by core modules.

        Additionally, each module may add its own beans in its private application context. For example Admin uses "context.central" but adds admin-specific beans. Tools use "context.tool" but add autopatch beans.

        2. Hibernate session factories
        Each tool had its own session factory. Core session factory had the knowledge of all tools' classes.

        Now private session factories has been removed. All tools and core services now use single, Core session factory. It knew about tools' hbm.xml files anyway. It might be the main reason of memory consumption decrease.

        3. Coupling
        New approach does not increase coupling between core and tools. Tools are still pluggable. Core "knows" as much about tools as it did before. And everything seems to be done in a smoother way.

        4. Tool deployment
        There is no need to append any information about tools' jars and application context config files to Central, Learning and Monioring any more. In fact, I have commented out this step in Tool Deployer and everything works fine. I have also removed two columns (context and classpath addition) from lams_tool table - we don't need them any more.

        5. Disadvantages
        Knowing all that, pitfalls might exist.
        Spring team writes about ContextSingletonBeanFactoryLocator:

        "use of this class and similar classes is unnecessary except (sometimes) for a small amount of glue code. Excessive usage will lead to code that is more tightly coupled, and harder to modify or test."

        "team strongly suggests that it be used sparingly and with caution. By far the vast majority of the code inside an application is best written in a Dependency Injection style, where that code is served out of a BeanFactory/ApplicationContext container, and has its own dependencies supplied by the container when it is created"

        "However, even such a singleton implementation sometimes has its use in the small glue layers of code that is sometimes needed to tie other code together"

        "As another example, in a complex J2EE app with multiple layers, with each layer having its own ApplicationContext definition (in a hierarchy), a class like SingletonBeanFactoryLocator may be used to demand load these contexts. "

        Till now beans of one tool were isolated one from another. They were using separate Hibernate session factories. And now everything is shared. We might need to test current architecture well to see if some new bugs won't show up.

        ================================
        JBoss

        1. Configuration files
        Guide:
        http://www.jboss.org/community/docs/DOC-10219
        and
        http://www.jboss.org/community/docs/DOC-9904

        It was possible to remove a good part of the described steps and still make LAMS work. Following services were removed

        a) JBoss Web Services

        b) RMI for Remote Invocations

        c) J2EE client deployer service

        d) HTTPInvoker

        e) JMX-Console

        f) scheduler-service.xml and JBoss Scheduler Manager

        g) JMS Queues

        h) CORBA/IIOP

        i) user-transaction-service.xml

        j) persistent MBean attributes

        k) RMI Classloading

        l) JNDIView MBean

        m) Pooled Invoker

        n) BeanShell deployer

        o) jboss-aop.deployer


        These services could not be removed:

        a) Hibernate session management services

        b) XA datasources

        c) JBossMQ

        d) Hypersonic

        e) client-side transaction management

        f) JBoss Naming locally

        2. Libraries
        It is possible to remove several libraries from JBoss default/lib directory. Some are bound with the configuration changes described before. We also don't use:

        autonumber-plugin.jar, bcel.jar

        Additionally, configuration was changed so we don't copy JBoss native JARs (jboss-cache, jgroups) into our lams.ear. Those JARs are either already available in JBoss, or we copy them into default/lib directory during deployment. There is no point in putting them into lams.ear.

        3. Results
        The amount of memory taken during start up is almost exactly the same (+/- 5MB) as before.

        What was removed are the idle services. Even if they are loaded into memory, they take minimal space and are inactive, thus removing them gave us virtually nothing.

        The most space is taken by the services that we are actually using. JBoss minimal server already takes 40MB and it has completely nothing on it. Most of this memory is taken by JRE. Our (default) server takes 60MB more, but we use Tomcat, different deployers, Hibernate support, security, JNDI, Hypersonic, parsers etc. Those services have to take some memory.

        Most probably there are ways to tune them so they use slightly less memory or work better. These changes won't impact start up, but rather the later operation of LAMS, for example when server is put under stress from lots of users.

        Removing libraries is also rather pointless. If we need a library, removing it will trigger an error. If don't need it, it won't be loaded whole into memory, but only selected classes.

        Most of LAMS memory is taken by Hibernate session factory. Maybe there is an option to make some mapped classes to be loade on demand only (lazy init), but sooner or later project will need them and they will be loaded anyway. It's just matter of time.
        Same goes for some other services that we could mark as lazy-loaded. It would give developers, some speed up when we are using only partial functionality to test things, but if we are talking about real live servers, they will eventually load all the services.

        4. Changes
        Ant tasks in lams_build/build.xml were added:
        "slim-jboss" creates a backup; then overwrites the configuration files with the new ones and removes some libraries and dirs from deployment directory
        "slim-jboss-revert" returns JBoss to the state before "slim-jboss" execution
        Show
        marcin Marcin Cieslak added a comment - Explanation of implemented changes: ================================================ Spring 1. Common application context All tools, admin, learning, monitoring etc. are Web applications. They might be modules of a single LAMS EAR, but for JBoss they are simply deployable WARs. It also means that they have separate application context. For each Web applications there is a root context, which is defined in our applicationContext.xml files. This context is common for all servlets defined in a Web application. Each servlet has also its own private context, but we weren't using this feature much. We were just using the shared context. Each of our tools (also learning, monitoring etc.) was importing some of application context config files from Central, Common etc. containing common beans like user management service. Also, each tool was adding some of its tool-specific beans. Event though those common beans were supposed to be shared and were declared as singletons, each of the tools' applications contexts were isolated from each other. This means each tool created its OWN SET OF SERVICES. Each common service and DAO had a copy in each tool. Additionally, our core services were importing all of our tools' application context files. Imports were added during tool deployment by appending lines into XML files in central, learning and monitoring. Spring has a class named ContextSingletonBeanFactoryLocator. This class allows searching for special configuration files (default: beanRefContext.xml) placed in classpath. These files contain bean definitions. Each bean is an application context. Each bean points to applicationContext.xml files that create this context. Beans can use inheritance. Locator gathers all special files it can find, glues them together and instantiates the beans (application contexts and beans within). After that these application context can be used as parents for Web application's root contexts. This means that each Web application (like one of our tools) can obtain such common application context and make it a parent of its root context, thus gaining access to all the beans. Those parent application context are instantiated only once, no matter which Web application calls for them (this is the "Singleton" word in class' name). 2 common application contexts have been created: "context.tool" - for tools' use. As tools don't use all the core services, this context defines only the required ones plus all of the private tools' contexts. Tools' contexts are searched for in the classpath automatically, using ant-style parts. This mechanism locates and adds all the tools' applicationContext.xml files. "context.central" - inherits from above. It adds all the remaining services like monitoring. It is used by core modules. Additionally, each module may add its own beans in its private application context. For example Admin uses "context.central" but adds admin-specific beans. Tools use "context.tool" but add autopatch beans. 2. Hibernate session factories Each tool had its own session factory. Core session factory had the knowledge of all tools' classes. Now private session factories has been removed. All tools and core services now use single, Core session factory. It knew about tools' hbm.xml files anyway. It might be the main reason of memory consumption decrease. 3. Coupling New approach does not increase coupling between core and tools. Tools are still pluggable. Core "knows" as much about tools as it did before. And everything seems to be done in a smoother way. 4. Tool deployment There is no need to append any information about tools' jars and application context config files to Central, Learning and Monioring any more. In fact, I have commented out this step in Tool Deployer and everything works fine. I have also removed two columns (context and classpath addition) from lams_tool table - we don't need them any more. 5. Disadvantages Knowing all that, pitfalls might exist. Spring team writes about ContextSingletonBeanFactoryLocator: "use of this class and similar classes is unnecessary except (sometimes) for a small amount of glue code. Excessive usage will lead to code that is more tightly coupled, and harder to modify or test." "team strongly suggests that it be used sparingly and with caution. By far the vast majority of the code inside an application is best written in a Dependency Injection style, where that code is served out of a BeanFactory/ApplicationContext container, and has its own dependencies supplied by the container when it is created" "However, even such a singleton implementation sometimes has its use in the small glue layers of code that is sometimes needed to tie other code together" "As another example, in a complex J2EE app with multiple layers, with each layer having its own ApplicationContext definition (in a hierarchy), a class like SingletonBeanFactoryLocator may be used to demand load these contexts. " Till now beans of one tool were isolated one from another. They were using separate Hibernate session factories. And now everything is shared. We might need to test current architecture well to see if some new bugs won't show up. ================================ JBoss 1. Configuration files Guide: http://www.jboss.org/community/docs/DOC-10219 and http://www.jboss.org/community/docs/DOC-9904 It was possible to remove a good part of the described steps and still make LAMS work. Following services were removed a) JBoss Web Services b) RMI for Remote Invocations c) J2EE client deployer service d) HTTPInvoker e) JMX-Console f) scheduler-service.xml and JBoss Scheduler Manager g) JMS Queues h) CORBA/IIOP i) user-transaction-service.xml j) persistent MBean attributes k) RMI Classloading l) JNDIView MBean m) Pooled Invoker n) BeanShell deployer o) jboss-aop.deployer These services could not be removed: a) Hibernate session management services b) XA datasources c) JBossMQ d) Hypersonic e) client-side transaction management f) JBoss Naming locally 2. Libraries It is possible to remove several libraries from JBoss default/lib directory. Some are bound with the configuration changes described before. We also don't use: autonumber-plugin.jar, bcel.jar Additionally, configuration was changed so we don't copy JBoss native JARs (jboss-cache, jgroups) into our lams.ear. Those JARs are either already available in JBoss, or we copy them into default/lib directory during deployment. There is no point in putting them into lams.ear. 3. Results The amount of memory taken during start up is almost exactly the same (+/- 5MB) as before. What was removed are the idle services. Even if they are loaded into memory, they take minimal space and are inactive, thus removing them gave us virtually nothing. The most space is taken by the services that we are actually using. JBoss minimal server already takes 40MB and it has completely nothing on it. Most of this memory is taken by JRE. Our (default) server takes 60MB more, but we use Tomcat, different deployers, Hibernate support, security, JNDI, Hypersonic, parsers etc. Those services have to take some memory. Most probably there are ways to tune them so they use slightly less memory or work better. These changes won't impact start up, but rather the later operation of LAMS, for example when server is put under stress from lots of users. Removing libraries is also rather pointless. If we need a library, removing it will trigger an error. If don't need it, it won't be loaded whole into memory, but only selected classes. Most of LAMS memory is taken by Hibernate session factory. Maybe there is an option to make some mapped classes to be loade on demand only (lazy init), but sooner or later project will need them and they will be loaded anyway. It's just matter of time. Same goes for some other services that we could mark as lazy-loaded. It would give developers, some speed up when we are using only partial functionality to test things, but if we are talking about real live servers, they will eventually load all the services. 4. Changes Ant tasks in lams_build/build.xml were added: "slim-jboss" creates a backup; then overwrites the configuration files with the new ones and removes some libraries and dirs from deployment directory "slim-jboss-revert" returns JBoss to the state before "slim-jboss" execution
        Hide
        marcin Marcin Cieslak added a comment -
        Everything seems to be working fine.
        Show
        marcin Marcin Cieslak added a comment - Everything seems to be working fine.
        marcin Marcin Cieslak made changes -
        Field Original Value New Value
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        ernieg Ernie Ghiglione added a comment -
        Wonderful stuff! This is the happiest jira closing in LAMS! ;-)
        Show
        ernieg Ernie Ghiglione added a comment - Wonderful stuff! This is the happiest jira closing in LAMS! ;-)
        ernieg Ernie Ghiglione made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            marcin Marcin Cieslak
            Reporter:
            marcin Marcin Cieslak
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development