cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

AyushModi038
by New Contributor III
  • 8451 Views
  • 7 replies
  • 9 kudos

Library installation in cluster taking a long time

I am trying to install "pycaret" libraray in cluster using whl file.But it is creating conflict in the dependency sometimes (not always, sometimes it works too.) ​My questions are -1 - How to install libraries in cluster only single time (Maybe from ...

  • 8451 Views
  • 7 replies
  • 9 kudos
Latest Reply
Spencer_Kent
New Contributor III
  • 9 kudos

@Retired_modWhat about question #1, which is what subsequent comments to this thread have been referring to? To recap the question: is it possible for "cluster-installed" libraries to be cached in such a way that they aren't completely reinstalled ev...

  • 9 kudos
6 More Replies
cpelazza
by New Contributor III
  • 2650 Views
  • 2 replies
  • 2 kudos

Resolved! Create a SQL (Python) UDF in a Serverless SQL Warehouse using an external library

Hi There,Scenario at work is as described in the subject line:Is it possible to author a SQL (Python) scalar UDF IN A SQL SERVERLESS WAREHOUSE, which involves a library NOT included in any Databricks Runtime? And how would one go about it? (Documenta...

  • 2650 Views
  • 2 replies
  • 2 kudos
grazie
by Contributor
  • 5459 Views
  • 5 replies
  • 0 kudos

Resolved! Is LEGACY_SINGLE_USER_STANDARD no longer supported?

Based on [an answer in this community](https://community.databricks.com/s/question/0D58Y00009jfKM1SAM/error-no-metastore-assigned-for-the-current-workspace), we set "data security mode" to LEGACY_SINGLE_USER_STANDARD in a cluster policy which needed...

  • 5459 Views
  • 5 replies
  • 0 kudos
Latest Reply
grazie
Contributor
  • 0 kudos

...and it works again - no changes made by us

  • 0 kudos
4 More Replies
lovinchanglei
by New Contributor II
  • 456 Views
  • 0 replies
  • 0 kudos

Failed to signup community version

I've been trying to create Community Edition account, but keep getting: "An error has occurred. Please try again later" message. I searched the other posts, there are some people running into the same issue as well, one post talked about VPN and succ...

lovinchanglei_0-1720155596817.png
Data Engineering
community edition
community version
signup
  • 456 Views
  • 0 replies
  • 0 kudos
Nino2024
by New Contributor II
  • 754 Views
  • 1 replies
  • 1 kudos

Migration Azure to GCP

is there a place I could find the best practices to migrate Databricks from Azure to GCP?

  • 754 Views
  • 1 replies
  • 1 kudos
Latest Reply
amr
Databricks Employee
  • 1 kudos

If you want to migrate Databricks to Databricks, you should use Databricks Teraform Exporter for that job

  • 1 kudos
RushabhNovelis
by New Contributor
  • 799 Views
  • 1 replies
  • 0 kudos

Is it possible to use a finetuned model with genie spaces feature?

I was trying to explore Genie room, I observed that I may need to finetune the model. Can I finetuned exisiting model with the same UI?

  • 799 Views
  • 1 replies
  • 0 kudos
Latest Reply
amr
Databricks Employee
  • 0 kudos

No. Genie spaces uses a fine tuned specialised text-2-sql model. fine-tuning text to 2 SQL model is not an easy job, so I would not recommend you doing yourself unless your business requires it.

  • 0 kudos
mayank_gupta
by New Contributor II
  • 913 Views
  • 2 replies
  • 0 kudos

Trying to create external table in Hive Metastore

Receiving this error: KeyProviderException: Failure to initialize configuration for storage account adlspersonal.dfs.core.windows.net: Invalid configuration value detected for fs.azure.account.keyI used hive meta store to save my table %python spark....

  • 913 Views
  • 2 replies
  • 0 kudos
Latest Reply
amr
Databricks Employee
  • 0 kudos

not sure about that error, but try to do it using SQL, see if it will workcreate table hive_metastore.annual_enterprise_survey as select * from catalog.defaul.table

  • 0 kudos
1 More Replies
AhmedAlnaqa
by Contributor
  • 1338 Views
  • 1 replies
  • 1 kudos

Resolved! Enhancements: interact with DBFS breadcrumbs

Hi there,This is my first thread and it's a baby-foot step in the Databricks community, especially Data engineering section.am working in the community edition and I found this enhancement needed to be implemented: The need is to make the breadcrumbs...

DBFS.png
  • 1338 Views
  • 1 replies
  • 1 kudos
Latest Reply
amr
Databricks Employee
  • 1 kudos

Good feedback, thank you. we are actully looking to complelty revamp the databricks community edition and the experience will be much simpler. stay tuned.

  • 1 kudos
Sid_SBA
by New Contributor
  • 643 Views
  • 1 replies
  • 0 kudos

Resolved! How to integrate the CI/CD process with Databricks using Azure Devops on Catalog level.

How to integrate the CI/CD process with Databricks using Azure Devops on Catalog level instead of workspace level. I would like to understand the process if this is possible, given that if the catalog is used in different workspaces in same subscript...

  • 643 Views
  • 1 replies
  • 0 kudos
Latest Reply
amr
Databricks Employee
  • 0 kudos

CICD is not related to catalogs, it is related to environment (workspaces), there are lots of tutorials on youtube on how to setup Azure DevOps CICD to move assets from one workspace to another and start a job. you will need to use Databricks Plugin ...

  • 0 kudos
anoopdk
by New Contributor
  • 1240 Views
  • 1 replies
  • 0 kudos

Add option to skip or deactivate a task

It would be beneficial to have an option like a toggle to activate or deactivate a Task in the Job graph interface. This mainly helps to skip execution of a task and reactivate it as required. Currently there is no option to say I want this task to b...

  • 1240 Views
  • 1 replies
  • 0 kudos
Latest Reply
amr
Databricks Employee
  • 0 kudos

Maybe just load the task with an empty notebook, and once decided just load the right notebook. not ideal but should do the job I guess

  • 0 kudos
Priya_Data_Eng
by New Contributor
  • 586 Views
  • 1 replies
  • 0 kudos

Special character data preservation

This data frame has two columns name and info. Name has value as John and info has vale as 1® VOC.After writing this data, I can read correct values in databricks but when I download the csv file and load it in notepad ( utf-8 ) , it is showing no va...

  • 586 Views
  • 1 replies
  • 0 kudos
Latest Reply
amr
Databricks Employee
  • 0 kudos

Try to read the file back into databricks using spark.read, do you the see the charchaters showing? if yes, then it is an editor problem, use another editor such as Notepad++, if not, then the data is not on the write encoder, try different encoder o...

  • 0 kudos
alesventus
by Contributor
  • 1019 Views
  • 2 replies
  • 0 kudos

Tasks in job are in pending state

I have databricks job with around 70 notebooks. When job starts, only one notebook gets executed and the rest of the notebooks that are at the beginning of the line are in state PENDING (not blocked). Looks like notebooks cannot run in parallel for t...

job_start.jpg job_middle.jpg
  • 1019 Views
  • 2 replies
  • 0 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 0 kudos

Maybe something related to autoscaling options? So when databricks detects increased workload it will scale up number of workers and then the rest of notebooks get executed. Do you use DLT ?

  • 0 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels