- 10460 Views
- 4 replies
- 2 kudos
Databricks asset bundles passing parameters using bundle run which are not declared
Hi,We recently decided to move to databricks asset bundles, one scenario that we are dealing with is we have different parameters passed to the same job which are handled in the notebook. With bundles when I try to pass parameters at runtime(which ar...
- 10460 Views
- 4 replies
- 2 kudos
- 2 kudos
Following syntax can be used.databricks bundle run -t ENV --params Param1=Value1,Param2=Value2 Job_NameJob Definition Parameter may look like this.
- 2 kudos
- 1793 Views
- 2 replies
- 0 kudos
Hi i have uploaded excel file in databricks but it shows different language.
Hi,I have attach one excel file in data bricks edition but unfortunately it shows a diiferent langaue in ouput whice i wrote display(df). below im attaching the screenshot please let me now thanking you in advance.
- 1793 Views
- 2 replies
- 0 kudos
- 0 kudos
CSV and Excel are not the same datatype. You can load the excel data into a pandas dataframe and then convert it to a pyspark dataframe.first, you have to install the openpyxl library %pip install openpyxl Then import PySpark Pandas: import pyspark.p...
- 0 kudos
- 6090 Views
- 4 replies
- 4 kudos
Resolved! Database creation error
I am trying to create database with external location abfss but facing the below error.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs....
- 6090 Views
- 4 replies
- 4 kudos
- 4 kudos
Changing it to a CLUSTER level for OAuth authentication helped me solve the problem.I wish the notebook AI bot could tell me the solution.before the changes, my configraiotn was at the notebook leve.and it has below errorsAnalysisException: org.apac...
- 4 kudos
- 2281 Views
- 4 replies
- 4 kudos
Resolved! code execution from Databrick folder
We are able to run a notebook that references Python code using import statements from a Databricks repowith the source code checked out. However, we encounter a ModuleNotFoundError when executing the same code from a folder.Error: ModuleNotFoundErro...
- 2281 Views
- 4 replies
- 4 kudos
- 4 kudos
Below worked as you have suggested.import os, sysproject_path = os.path.abspath("/Workspace/<folder-name-1>/<folder-name-2>/<top-level-code-folder>")if project_path not in sys.path: sys.path.append(project_path)
- 4 kudos
- 2461 Views
- 3 replies
- 0 kudos
S3 Data access through unity
Hey All I have the following issue : I've connected a s3 bucket through unity catalog as an external source. I perfectly see the files of my s3 bucket when i scroll through the catalog using the user interface. However when I try to connect through a...
- 2461 Views
- 3 replies
- 0 kudos
- 0 kudos
Hey Chandra thank you for your answer. The path is a volume path indeed, : /Volumes/my_path_in_volume
- 0 kudos
- 9723 Views
- 6 replies
- 0 kudos
Resolved! SQLServerException: deadlock
I'm using databricks to connect to a SQL managed instance via JDBC. SQL operations I need to perform include DELETE, UPDATE, and simple read and write. Since spark syntax only handles simple read and write, I had to open SQL connection using Scala an...
- 9723 Views
- 6 replies
- 0 kudos
- 0 kudos
@swzzzsw Since you are performing database operations, to reduce the chances of deadlocks, make sure to wrap your SQL operations inside transactions using commit and rollback.Another approachs to consider is adding retry logic or using Isolation Leve...
- 0 kudos
- 4222 Views
- 1 replies
- 0 kudos
External Table From BigQuery
I'm working on implementing Unity Catalog, and part of that is determining how to handle our BigQuery tables. We need to utilize them to connect to another application, or else we'd stay within regular delta tables on Databricks.The page https://docs...
- 4222 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Kayla,I know your question is already a year old, but it's possible to create BigQuery tables in Unity Catalog: https://docs.gcp.databricks.com/en/query-federation/bigquery.htmlBest regardsLorenz
- 0 kudos
- 1510 Views
- 1 replies
- 0 kudos
How can I effectively integrate my ai laptop’s local resources with Databricks for AI model training
Hi everyone, I’m currently working on AI projects and using an ai laptop for local development, while leveraging Databricks for larger model training and experimentation. I’m looking for advice on how to effectively integrate my ai laptop local resou...
- 1510 Views
- 1 replies
- 0 kudos
- 0 kudos
Those are a lot of questions.For each of your questions, one could come up with an answer that runs locally on your computer.But......you will probably regret this as it will be very hard to maintain and deploy to databricks.It won't be an exact copy...
- 0 kudos
- 3347 Views
- 1 replies
- 0 kudos
Upset Plot in Databricks
I am trying to create an Upset Plot using following code, but my output is not as expected. See Image1: Output which I am getting vs Image2: Output expected. Where the total count of overlap is displayed at the top of each bar, Bar size is proportion...
- 3347 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @PJ11 ,As per documentation: UpSetPlot internally works with data based on Pandas data structures: a Series when all you care about is counts, or a DataFrame when you’re interested in visualising additional properties of the data, such as with the...
- 0 kudos
- 1534 Views
- 2 replies
- 4 kudos
Resolved! Cannot be able to find DBFS under settings->Advance in the Databricks community edition.
Cannot be able to find DBFS under settings->Advance in the Databricks community edition.
- 1534 Views
- 2 replies
- 4 kudos
- 4231 Views
- 3 replies
- 0 kudos
DBFS option
I can't find dbfs option in my free data bricks community edition when I tried to see location of the table.
- 4231 Views
- 3 replies
- 0 kudos
- 972 Views
- 1 replies
- 0 kudos
FROM databricksruntime/standard:11.3-LTS as application
Below command was working until yesterday for 11.3-LTS base image. It started failing from today (17Oct 2024 11AM IST)FROM databricksruntime/standard:11.3-LTS as applicationRUN apt-get install -y python3-venv It throws the error :Reading package lis...
- 972 Views
- 1 replies
- 0 kudos
- 0 kudos
Any specific reason you are trying to containerize this? Can you try adding an apt-get update before python3-venv RUN apt-get update && \ apt-get install -y python3-venv
- 0 kudos
- 1003 Views
- 1 replies
- 0 kudos
How to transform of date to EDM
How to transform of date to EDM
- 1003 Views
- 1 replies
- 0 kudos
- 0 kudos
Can you please provide some more details about your question?
- 0 kudos
- 2019 Views
- 1 replies
- 0 kudos
Jenkins Alternatives for Data Pipeline Automation?
I’ve been managing data pipelines with Jenkins but would like to explore other options that might work better for data-centric projects. Has anyone tried GitLab CI or Azure DevOps for similar use cases? Looking for insights into what might offer bett...
- 2019 Views
- 1 replies
- 0 kudos
- 0 kudos
You can use Github Actions or Azure DevOps as an alternative.I use Azure DevOps Pipelines for my projects and have had very good experiences. You can find instructions on how to do the whole thing with Github Actions for example, in the official docu...
- 0 kudos
- 4045 Views
- 4 replies
- 1 kudos
Resolved! Get job run_id of run_job_task in an orchestration job
Hi,I have a Databricks job which orchestrates the run of two jobs: job1 and job2 using run_job_task.job2 depends on job1 and would need to use the run_id of job1 as a parameter. How can this be done?I see that you can only easily access the task run ...
- 4045 Views
- 4 replies
- 1 kudos
- 1 kudos
@MihaiTache You can achieve this by utilizing a combination of dbutils.widgets.get and dbutils.jobs.taskValues.set. The approach involves extracting the run_id from Job1 and passing it as a value to Job2 using taskValues.set. This allows seamless com...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
