- 7086 Views
- 3 replies
- 3 kudos
Resolved! Databricks community edition still available?
Is Databricks platform still available in its Community edition (outside Azure, AWS or GCP)? Can someone share the updated link?Thanks,Elisa
- 7086 Views
- 3 replies
- 3 kudos
- 3 kudos
Look : it is still available but afaik always linked to azure/gcp/aws
- 3 kudos
- 19881 Views
- 1 replies
- 1 kudos
Resolved! Notebook Langchain ModuleNotFoundError: No module named 'langchain.retrievers.merger_retriever'
Hi,As mentioned in the title, receiving this error despite%pip install --upgrade langchainSpecific line of code:from langchain.retrievers.merger_retriever import MergerRetriever All other langchain import works when this is commented out. Same line w...
- 19881 Views
- 1 replies
- 1 kudos
- 1 kudos
More specifically: langchain releases a new update every few days, and it is likely that you are using code or a library that needs a later version of langchain than you have (or, perhaps, a later version that removed whatever part of langchain you r...
- 1 kudos
- 2513 Views
- 2 replies
- 1 kudos
Databricks notebook execution using only one task
I am running a databricks notebook. While running, i only see one task on one worker getting started. My cluster has min 6 workers but seems like they are not getting used.I am performing a read operation from Cosmos DB.Can someone please help me her...
- 2513 Views
- 2 replies
- 1 kudos
- 1 kudos
If your code does not use Spark, it will not use any machines except the driver. If you're using Spark but your source data that you operate on has 1 partition, there will be only 1 task. Hard to say more without knowing what you are doing in more de...
- 1 kudos
- 6035 Views
- 2 replies
- 1 kudos
MIT License and Fine-tuning
Some questions related to fine-tuning and the MIT License, I read the MIT license but still confusing about some points.If I fine-tune the Dolly-v2 model, say using LoRA and my own dataset,Do I "own" the fine-tuned model?Am I allow to change the name...
- 6035 Views
- 2 replies
- 1 kudos
- 1 kudos
I am not sure I agree with the discussion so far. While none of here are lawyers, I think it's fairly straightforward to reason about the licensing.You have created a combined, derivative work from the Dolly weights in this case. You have copyright i...
- 1 kudos
- 1766 Views
- 0 replies
- 0 kudos
Databricks Rstudio Init Script Deprecated
OK so I'm trying to use Open Source Rstudio on Azure Databricks.I'm following the instructions here: https://learn.microsoft.com/en-us/azure/databricks/sparkr/rstudio#install-rstudio-server-open-source-editionI've installed the necessary init script ...
- 1766 Views
- 0 replies
- 0 kudos
- 2136 Views
- 2 replies
- 1 kudos
Getting FileNotFoundException while using cloudFiles
Hi,Following is the code i am using the ingest the data incrementally (weekly).val ssdf = spark.readStream.schema(schema) .format("cloudFiles").option("cloudFiles.format", "parquet").load(sourceUrl).filter(criteriaFilter)val transformedDf = ssdf.tran...
- 2136 Views
- 2 replies
- 1 kudos
- 8957 Views
- 6 replies
- 3 kudos
Resolved! Using databricks for end to end flow? rather than using ADF for extracting data
Currently, in our company we are using ADF+DATABRICKS for all batch integration. Using ADF first data is copied to ADLS gen 2 (from different sources like on prem servers, ftp solution file sharing, etc), then it is reformatted to csv and it is copie...
- 8957 Views
- 6 replies
- 3 kudos
- 3 kudos
@-werners- Is there any benefit of doing the extract part in databricks itself? Unlike our current architecture, where we first load to adls using adf. I guess it is worth doing all end to end using databricks if there is better processing, lower lat...
- 3 kudos
- 1645 Views
- 1 replies
- 0 kudos
RBAC, Security & Privacy controls
Could you please share us best practices on implementation of RBAC, Security & Privacy controls in Databricks
- 1645 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Could you please check on https://docs.databricks.com/en/lakehouse-architecture/security-compliance-and-privacy/best-practices.html and see if this helping? Also, please tag @Debayan with your next comment which will notify me. Thanks!
- 0 kudos
- 2174 Views
- 0 replies
- 0 kudos
Records are missing while creating new dataframe from one big dataframe using filter
Hi,I have data in file like belowI have different types of row in my input file, column number 8 defines the type of the record.In the above file we have 4 types of records 00 to 03My requirement is:There will be multiple files in the source path, ea...
- 2174 Views
- 0 replies
- 0 kudos
- 2806 Views
- 2 replies
- 0 kudos
Records are missing while creating new data from one big dataframe using filter
Hi,I have data in file like below I have different types of row in my input file, column number 8 defines the type of the record.In the above file we have 4 types of records 00 to 03My requirement is:There will be multiple files in the source path, e...
- 2806 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Retired_mod ,If i run again with same files sometimes records will be missed from same files of the previous run or records will be missed from different file.Example:run1: 1 record missing in file1, no issue with other filesrun2: 1 record missin...
- 0 kudos
- 3780 Views
- 0 replies
- 0 kudos
Dashboard backup/download
Hola all, I'm trying to download all the dashboard definitions, however, I can only download the folder structure with no file inside. The procedure I'm using is* Go to the dashboard folder* Download it as DBC or source archiveUnfortunately, the DBC ...
- 3780 Views
- 0 replies
- 0 kudos
- 1471 Views
- 1 replies
- 0 kudos
ganglia metrics
Hello Everyone, I have build this script in order to collect ganglia metrics but the size of stderr and sdtout ganglia is 0. It doesn't work. I Have put this script on Workspace due to migration databricks all init-script should be place on Workspace...
- 1471 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Is there any error you are getting? Also, please tag @Debayan with your next comment which will notify me. Thanks!
- 0 kudos
- 1723 Views
- 1 replies
- 0 kudos
GCP hosted Databricks - DBFS temp files - Not Found
I've been working on obtaining DDL at the schema level in Hive Metastore within GCP-hosted Databricks. I've implemented a Python code that generates SQL files in the dbfs/temp directory. However, when running the code, I'm encountering a "file path n...
- 1723 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, the error code snippet with the whole error may help to determine the issue, also, considering the above points may also work as a fix.
- 0 kudos
- 2729 Views
- 1 replies
- 0 kudos
Unable to find permission button in Sql Warehouse for providing Access
Hi Everyone, am unable to see the permission button in sql warehouse to provide access to other users.I have admin rights and databricks is premium subscription.
- 2729 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Could you please provide a screenshot of the SQL warehouse? Also, you can go through: https://docs.databricks.com/en/security/auth-authz/access-control/sql-endpoint-acl.htmlAlso, please tag @Debayan with your next comment which will notify me. Th...
- 0 kudos
- 1226 Views
- 1 replies
- 1 kudos
SQL Serverless - cost view
Hi,Anyone knows how I'm able to monitor cost of the SQL Serverless? I'm using Databricks in Azure and I'm not sure where to find cost generated by compute resources hosted on Databricks.
- 1226 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi, You can calculate the pricing in https://www.databricks.com/product/pricing/databricks-sql also, https://azure.microsoft.com/en-in/pricing/details/databricks/#:~:text=Sign%20in%20to%20the%20Azure,asked%20questions%20about%20Azure%20pricing. For A...
- 1 kudos
