!pip install llama-index!pip install llama-index-llms-groq!pip install llama-index-core!pip install llama-index-readers-file!pip install llama-index-tools-wolfram-alpha!pip install llama-index-embeddings-huggingface!pip install 'crewai[tools]'

from google.colab import userdatafrom llama_index.llms.groq import Groqgroq_api_key = userdata.get('GROQ_API_KEY')#llm = Groq(model="llama3-70b-8192", api_key=groq_api_key)#response = llm.complete("Explain the importance of low latency LLMs")print(response)
######################################Response############################Low-latency Large Language Models (LLMs) are crucial in various applications where real-time or near-real-time processing is essential. Here are some reasons why low-latency LLMs are important:
1. **Interactive Systems**: In interactive systems like chatbots, virtual assistants, and conversational AI, low-latency LLMs enable rapid response times, making the interaction feel more natural and human-like. This is particularly important in applications where users expect immediate responses, such as customer support or language translation.2. **Real-time Decision Making**: In applications like autonomous vehicles, robotics, or medical diagnosis, low-latency LLMs can process and analyze vast amounts of data in real-time, enabling swift decision-making and reaction to changing circumstances.3. **Live Streaming and Broadcasting**: Low-latency LLMs can facilitate real-time language translation, sentiment analysis, or content moderation in live streaming and broadcasting applications, enhancing the viewer experience and ensuring timely content delivery.4. **Gaming and Esports**: In online gaming and esports, low-latency LLMs can improve the gaming experience by enabling faster language processing, sentiment analysis, and chat moderation, reducing lag and enhancing overall performance.5. **Healthcare and Emergency Services**: In healthcare and emergency services, low-latency LLMs can quickly process medical records, diagnose conditions, and provide critical information to healthcare professionals, saving lives and improving patient outcomes.6. **Financial Trading and Analytics**: Low-latency LLMs can rapidly analyze large datasets, enabling high-frequency trading, sentiment analysis, and risk assessment in financial markets, helping traders and analysts make informed decisions.7. **Cybersecurity**: In cybersecurity, low-latency LLMs can quickly detect and respond to threats, such as malware, phishing attacks, or DDoS attacks, reducing the attack surface and minimizing damage.8. **Edge Computing and IoT**: As IoT devices proliferate, low-latency LLMs can process data closer to the source, reducing latency and improving real-time decision-making in applications like smart homes, cities, or industrial automation.9. **Accessibility and Inclusion**: Low-latency LLMs can enable real-time language translation, captioning, and transcription, improving accessibility and inclusion for people with disabilities, language barriers, or hearing impairments.10. **Competitive Advantage**: In many industries, low-latency LLMs can provide a competitive advantage by enabling faster decision-making, improved customer experiences, and increased operational efficiency, ultimately driving business success.
To achieve low latency in LLMs, researchers and developers are exploring various techniques, including:
1. Model pruning and knowledge distillation2. Quantization and precision reduction3. Parallel processing and distributed computing4. Edge computing and decentralized architectures5. Optimized hardware and software designs6. Caching and memoization7. Lazy loading and just-in-time compilation
By reducing latency in LLMs, we can unlock new possibilities in various applications, leading to improved user experiences, increased efficiency, and enhanced decision-making capabilities.

## Crewai requires a chat based model for bindingfrom langchain_openai import ChatOpenAI
chat_llm = ChatOpenAI(openai_api_base="https://api.groq.com/openai/v1",openai_api_key=groq_api_key,model="llama3-70b-8192",temperature=0,max_tokens=1000,)

!wget "https://s23.q4cdn.com/407969754/files/doc_financials/2019/ar/Uber-Technologies-Inc-2019-Annual-Report.pdf" -O uber_10k.pdf

from llama_index.core import SimpleDirectoryReader, VectorStoreIndexfrom llama_index.llms.openai import OpenAIimport osfrom langchain_openai import ChatOpenAI
#reader = SimpleDirectoryReader(input_files=["uber_10k.pdf"])docs = reader.load_data()docs[1]
##############################################################Document(id_='dd161725-2512-4b03-a689-accc69dc46d4', embedding=None, metadata={'page_label': '2', 'file_name': 'uber_10k.pdf', 'file_path': 'uber_10k.pdf', 'file_type': 'application/pdf', 'file_size': 2829436, 'creation_date': '2024-06-30', 'last_modified_date': '2020-03-31'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={}, text='69\nCountries\n10K+\nCities\n$65B\nGross Bookings\n111M\nMAPCs\n7B\nTripsA global tech \nplatform at \nmassive scale\nServing multiple multi-trillion \ndollar markets with products \nleveraging our core technology \nand infrastructure\nWe believe deeply in our bold mission. Every minute \nof every day, consumers and Drivers on our platform \ncan tap a button and get a ride or tap a button and \nget work. We revolutionized personal mobility with \nridesharing, and we are leveraging our platform to \nredefine the massive meal delivery and logistics \nindustries. The foundation of our platform is our \nmassive network, leading technology, operational \nexcellence, and product expertise. Together, these \nelements power movement from point A to point B.', mimetype='text/plain', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n')

from llama_index.embeddings.huggingface import HuggingFaceEmbedding
# loads BAAI/bge-small-en-v1.5embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")

index = VectorStoreIndex.from_documents(docs,embed_model=embed_model,)query_engine = index.as_query_engine(similarity_top_k=5, llm=llm)

from crewai_tools import LlamaIndexToolquery_tool = LlamaIndexTool.from_query_engine(query_engine,name="Uber 2019 10K Query Tool",description="Use this tool to lookup the 2019 Uber 10K Annual Report",)#query_tool.args_schema.schema()
########################################{'title': 'QueryToolSchema', 'description': 'Schema for query tool.', 'type': 'object', 'properties': {'query': {'title': 'Query', 'description': 'Search query for the query tool.', 'type': 'string'}}, 'required': ['query']}

import osfrom crewai import Agent, Task, Crew, Process
# Define your agents with roles and goalsresearcher = Agent(role="Senior Financial Analyst",goal="Uncover insights about different tech companies",backstory="""You work at an asset management firm.Your goal is to understand tech stocks like Uber.""",verbose=True,allow_delegation=False,tools=[query_tool],llm=chat_llm,)writer = Agent(role="Tech Content Strategist",goal="Craft compelling content on tech advancements",backstory="""You are a renowned Content Strategist, known for your insightful and engaging articles.You transform complex concepts into compelling narratives.""",llm=chat_llm,verbose=True,allow_delegation=False,)

# Create tasks for your agentstask1 = Task(description="""Conduct a comprehensive analysis of Uber's risk factors in 2019.""",expected_output="Full analysis report in bullet points",agent=researcher,)
task2 = Task(description="""Using the insights provided, develop an engaging blogpost that highlights the headwinds that Uber faces.Your post should be informative yet accessible, catering to a casual audience.Make it sound cool, avoid complex words.""",expected_output="Full blog post of at least 4 paragraphs",agent=writer,)

crew = Crew(agents=[researcher, writer],tasks=[task1, task2],verbose=2,# You can set it to 1 or 2 to different logging levels)