- 4053 Views
- 5 replies
- 0 kudos
Query Serverless SQL Warehouse from Spark Submit Job
I am trying to load data from a table in SQL warehouse using spark.sql("SELECT * FROM <table>") in a spark submit job, but the job is failing with [TABLE_OR_VIEW_NOT_FOUND] The table or view . The same statement is working in notebook but not in a jo...
- 4053 Views
- 5 replies
- 0 kudos
- 0 kudos
- when you query table manually and running job - do both those actions happens in same Databricks Workspace- what is job configuration - who is job Owner or Run As Account -> do this principal/persona has access to the table ?
- 0 kudos
- 3344 Views
- 2 replies
- 0 kudos
Convert pdf's is into structured data
Is there anything on Databricks to help read PDF (payment invoices and receipts for example) and convert it to structured data?
- 3344 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks! Converting PDF format is sometimes a difficult task as not all converters provide accuracy. I want to share with you one interesting tool I recently discovered that can make your work even more efficient. I recently came across an amazing onl...
- 0 kudos
- 6418 Views
- 3 replies
- 0 kudos
Resolved! Error on Starting Databricks SQL Warehouse Serverless with Instance Profile
I have two workspaces, one in us-west-2 and the other in ap-southeast-1. I have configured the same instance profile for both workspaces. I followed the documentation to set up the instance profile for Databricks SQL Warehouse Serverless by adding th...
- 6418 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Tam , Hope you are doing well! I checked the error in details and it would be because the Instance Profile Name and the Role ARN name don't match exactly. Please see points 3 and 4 here in the docs: https://docs.databricks.com/sql/admin/serverle...
- 0 kudos
- 3285 Views
- 1 replies
- 0 kudos
CDC DLT
Hi all,I would appreciate some clarity regarding the DLT and CDC. So my first question would be, when it comes to the "source" table in the synta, is that CDC table or? Further, if we want to use only databricks, would mounting foreign catalog be a g...
- 3285 Views
- 1 replies
- 0 kudos
- 8188 Views
- 4 replies
- 1 kudos
Rewrite Notebooks Programatically
Hello,I want to refactor the notebook programatically. So, written the code as follows: import requestsimport base64# Databricks Workspace API URLsworkspace_url = f"{host}/api/2.0/workspace"export_url = f"{workspace_url}/export"import_url = f"{worksp...
- 8188 Views
- 4 replies
- 1 kudos
- 3298 Views
- 1 replies
- 0 kudos
Databricks Error while executing this line of code
import geopandas as gpdfrom shapely.geometry import *Pd_csv_sel_pq_gg = gpd.GeoDataFrame(Points_csv_sel_pq_gg.toPandas(), geometry="geometry") Error is given below /databricks/spark/python/pyspark/sql/pandas/utils.py:37: DeprecationWarning: distutil...
- 3298 Views
- 1 replies
- 0 kudos
- 3642 Views
- 2 replies
- 1 kudos
Processing Notebook in python
Hi,I exported notebook from my workspace into my local machine and want to read it in my python code .Is there a way to read the content of my notebook programmatically and make necessary changes and save as dbc/html notebook.
- 3642 Views
- 2 replies
- 1 kudos
- 1 kudos
Not sure what you are trying to accomplish here. If you want to export a notebook as python to do manual editing locally, and then import it back into your workspace why not use repos and connect to it using VSCode etc? You can export the notebook as...
- 1 kudos
- 11688 Views
- 5 replies
- 1 kudos
Dash in Databricks notebook directly
Hi team,Is there a way to embed plotly dash directly inside Databricks notebook?Thanks
- 11688 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @MikeGo - Unfortunately, it's not possible today to embed Dash in a Databricks notebook cell without our Enterprise-level databricks-dash library. Longer term, we are working towards Dash natively working within Databricks notebooks, but that time...
- 1 kudos
- 2513 Views
- 0 replies
- 0 kudos
Foreign Catalog SQL Server Dynamic Port
When creating a Foreign Catalog SQL Server Connection, a port number is required. However, many sql servers have dynamic ports and the port number keeps changing. Is there a solution for this?In most common cases, it should allow instance name instea...
- 2513 Views
- 0 replies
- 0 kudos
- 11380 Views
- 2 replies
- 0 kudos
Spark submit - not reading one of my --py-files arguments
Hi. In Databricks workflows, I submit a spark job (Type = "Spark Submit"), and a bunch of parameters, starting with --py-files.This works where all the files are in the same s3 path, but I get errors when I put a "common" module in a different s3 pat...
- 11380 Views
- 2 replies
- 0 kudos
- 0 kudos
This below is catered for yarn modeif your application code primarily consists of Python files and does not require a separate virtual environment with specific dependencies, you can use the --py-files argument in spark-submitspark-submit --verbose ...
- 0 kudos
- 2349 Views
- 0 replies
- 0 kudos
How to set a job trigger in a yml deployment asset bundle?
Working on an asset bundle/yml file to deploy a job and some notebooks. How to specify within the yml file a trigger to run the job when files arrive at a specified location? Thanks in advance!
- 2349 Views
- 0 replies
- 0 kudos
- 12987 Views
- 5 replies
- 2 kudos
Resolved! Databricks Job: Package Name and EntryPoint parameters for the Python Wheel file
I have created Python wheel file with simple file structure and uploaded into cluster library and was able to run the packages in Notebook but, when I am trying to create a Job using python wheel and provide the package name and run the task it fails...
- 12987 Views
- 5 replies
- 2 kudos
- 2 kudos
There you can see a complete template project with (the new!!!) Databricks Asset Bundles tool and a python wheel task. Please, follow the instructions for deployment.https://github.com/andre-salvati/databricks-template
- 2 kudos
- 5467 Views
- 2 replies
- 0 kudos
databricks asset bundles - error demo project
Hi,I am using the v0.205.0 version of the CLI. I wanted to test the demo project (databricks bundle init) of the Databricks Asset Bundles, however I am getting an error after databricks bundle deploy (validate is ok). artifacts.whl.AutoDetect: Detec...
- 5467 Views
- 2 replies
- 0 kudos
- 0 kudos
There you can see a complete template project with Databricks Asset Bundles and a python wheel task. Please, follow the instructions for deployment.https://github.com/andre-salvati/databricks-template
- 0 kudos
- 4124 Views
- 2 replies
- 1 kudos
Resolved! [BUG] Databricks install WHL as JAR in Python Wheel Task?
I'm using Python Wheel Task in Databricks job with WHEEL dependencies. However, the cluster installed the dependencies as JAR instead of WHEEL. Is this an expected behavior or a bug?
- 4124 Views
- 2 replies
- 1 kudos
- 1 kudos
There you can see a complete template project with a python wheel task and Databricks Asset Bundles. Please, follow the instructions for deployment.https://github.com/andre-salvati/databricks-template
- 1 kudos
- 10497 Views
- 3 replies
- 0 kudos
Databricks Tasks Python wheel : How access to JobID & runID ?
I'm using Python (as Python wheel application) on Databricks.I deploy & run my jobs using dbx.I defined some Databricks Workflow using Python wheel tasks.Everything is working fine, but I'm having issue to extract "databricks_job_id" & "databricks_ru...
- 10497 Views
- 3 replies
- 0 kudos
- 0 kudos
There you can see a complete template project with Databricks Asset Bundles and python wheel task. Please, follow the instructions for deployment.https://github.com/andre-salvati/databricks-templateIn particular, take a look at the workflow definitio...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 576 | |
| 349 | |
| 287 |
