cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Policepatil
by New Contributor III
  • 6023 Views
  • 0 replies
  • 0 kudos

Is it good to process files in multithreading?

Hi,I need to process nearly 30 files from different locations and insert records to RDS.I am using multi-threading to process these files parallelly like below. def process_files(file_path):    <process files here>    1. Find bad records based on fie...

  • 6023 Views
  • 0 replies
  • 0 kudos
bachan
by New Contributor II
  • 1560 Views
  • 1 replies
  • 0 kudos

Data Insertion

Scenario: Data from blob storage to SQL db once a week.I have 15(from current date to next 15 days) days data into the blob storage, stored date wise in parquet format, and after seven days the next 15 days data will be inserted. Means till 7th day t...

  • 1560 Views
  • 1 replies
  • 0 kudos
Sivaji
by New Contributor
  • 913 Views
  • 1 replies
  • 0 kudos

Databricks data engineer associate Exam got suspended.

Hello Team, I encountered Pathetic experience while attempting my 1st DataBricks certification. Abruptly, Proctor asked me to show my desk, after showing he/she asked multiple times.. wasted my time and then suspended my exam. I want to file a compla...

Community Platform Discussions
Data engineer Associate
Exam.
  • 913 Views
  • 1 replies
  • 0 kudos
Latest Reply
Cert-Team
Databricks Employee
  • 0 kudos

Hi @Sivaji Sorry to hear you had a bad experience, and that you got a slow response here in the community. I see that you have taken and passed the exam, Congratulations!For the future, our support team handles cases from here first so it tends to be...

  • 0 kudos
JRL
by New Contributor II
  • 1487 Views
  • 1 replies
  • 0 kudos

Github "Danger Zone"

There is a "Danger zone" appearing in the Github indicating that the repositories I share on Databricks should be Suspended and possibly that Databricks should be uninstalled.   This may be something standard in Github.  Has anyone run across it?

DangerZone.PNG
  • 1487 Views
  • 1 replies
  • 0 kudos
Latest Reply
sean_owen
Databricks Employee
  • 0 kudos

It's not telling you that you should do these things. It's telling you that you may break stuff by doing these things. Yes the "Danger Zone" is a thing on Github, it tries to warn you before you do things like click to delete a repo.

  • 0 kudos
jermaineharsh
by New Contributor III
  • 851 Views
  • 0 replies
  • 1 kudos

How to switch from free trial to Community Edition of Databricks in my Azure workspace?

hello,I am trying to switch into Databricks Community Edition after a 14 day trial. I was able to register, but when I try to start my new cluster, I get an error message, "Cluster start feature is currently disabled, and the cluster does not run".In...

  • 851 Views
  • 0 replies
  • 1 kudos
Picci
by New Contributor III
  • 3939 Views
  • 3 replies
  • 3 kudos

Resolved! Databricks community edition still available?

Is Databricks platform still available in its Community edition (outside Azure, AWS or GCP)? Can someone share the updated link?Thanks,Elisa

  • 3939 Views
  • 3 replies
  • 3 kudos
Latest Reply
jamescw
New Contributor II
  • 3 kudos

Look : it is still available but afaik always linked to azure/gcp/aws

  • 3 kudos
2 More Replies
bento
by New Contributor
  • 14147 Views
  • 1 replies
  • 1 kudos

Resolved! Notebook Langchain ModuleNotFoundError: No module named 'langchain.retrievers.merger_retriever'

Hi,As mentioned in the title, receiving this error despite%pip install --upgrade langchainSpecific line of code:from langchain.retrievers.merger_retriever import MergerRetriever All other langchain import works when this is commented out. Same line w...

  • 14147 Views
  • 1 replies
  • 1 kudos
Latest Reply
sean_owen
Databricks Employee
  • 1 kudos

More specifically: langchain releases a new update every few days, and it is likely that you are using code or a library that needs a later version of langchain than you have (or, perhaps, a later version that removed whatever part of langchain you r...

  • 1 kudos
ankit_batra1
by New Contributor
  • 1660 Views
  • 2 replies
  • 1 kudos

Databricks notebook execution using only one task

I am running a databricks notebook. While running, i only see one task on one worker getting started. My cluster has min 6 workers but seems like they are not getting used.I am performing a read operation from Cosmos DB.Can someone please help me her...

  • 1660 Views
  • 2 replies
  • 1 kudos
Latest Reply
sean_owen
Databricks Employee
  • 1 kudos

If your code does not use Spark, it will not use any machines except the driver. If you're using Spark but your source data that you operate on has 1 partition, there will be only 1 task. Hard to say more without knowing what you are doing in more de...

  • 1 kudos
1 More Replies
FutureLegend
by New Contributor III
  • 3340 Views
  • 2 replies
  • 1 kudos

MIT License and Fine-tuning

Some questions related to fine-tuning and the MIT License, I read the MIT license but still confusing about some points.If I fine-tune the Dolly-v2 model, say using LoRA and my own dataset,Do I "own" the fine-tuned model?Am I allow to change the name...

  • 3340 Views
  • 2 replies
  • 1 kudos
Latest Reply
sean_owen
Databricks Employee
  • 1 kudos

I am not sure I agree with the discussion so far. While none of here are lawyers, I think it's fairly straightforward to reason about the licensing.You have created a combined, derivative work from the Dolly weights in this case. You have copyright i...

  • 1 kudos
1 More Replies
meystingray
by New Contributor II
  • 1384 Views
  • 0 replies
  • 0 kudos

Databricks Rstudio Init Script Deprecated

OK so I'm trying to use Open Source Rstudio on Azure Databricks.I'm following the instructions here: https://learn.microsoft.com/en-us/azure/databricks/sparkr/rstudio#install-rstudio-server-open-source-editionI've installed the necessary init script ...

  • 1384 Views
  • 0 replies
  • 0 kudos
Ruby8376
by Valued Contributor
  • 5547 Views
  • 6 replies
  • 3 kudos

Resolved! Using databricks for end to end flow? rather than using ADF for extracting data

Currently, in our company we are using ADF+DATABRICKS for all batch integration. Using ADF first data is copied to ADLS gen 2 (from different sources like on prem servers, ftp solution file sharing, etc), then it is reformatted to csv and it is copie...

  • 5547 Views
  • 6 replies
  • 3 kudos
Latest Reply
Ruby8376
Valued Contributor
  • 3 kudos

@-werners- Is there any benefit of doing the extract part in databricks itself? Unlike our current architecture, where we first load to adls using adf. I guess it is worth doing all end to end using databricks if there is better processing, lower lat...

  • 3 kudos
5 More Replies
Policepatil
by New Contributor III
  • 1695 Views
  • 0 replies
  • 0 kudos

Records are missing while creating new dataframe from one big dataframe using filter

Hi,I have data in file like belowI have different types of row in my input file, column number 8 defines the type of the record.In the above file we have 4 types of records 00 to 03My requirement is:There will be multiple files in the source path, ea...

Policepatil_0-1693826562540.png Policepatil_1-1693826571781.png Policepatil_2-1693826609156.png Policepatil_3-1693826641543.png
  • 1695 Views
  • 0 replies
  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Top Kudoed Authors