- 4001 Views
- 2 replies
- 1 kudos
Resolved! error installing the igraph and networkD3 library
Hi!I am trying to install the igraph and networkD3 CRAN packages for use within a notebook. However, I am receiving the below installation error when attempting to do so.Could someone please assist? Thank you!* installing *source* package ‘igraph’ .....
- 4001 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi! I just figured this out myself. I'm not sure why this is suddenly occurring, since igraph has always loaded fine for me in databricks but didn't this week. I found that the following solution worked.In your notebook before installing your R libra...
- 1 kudos
- 4576 Views
- 2 replies
- 4 kudos
How to write log entries from a Delta Live Table pipeline.
From a notebook I can import the log4j logger from cs and write to a log like so:log4jLogger = sc._jvm.org.apache.log4jLOGGER = log4jLogger.LogManager.getLogger(__name__)LOGGER.info("pyspark script logger initialized")But this does not work in a Delt...
- 4576 Views
- 2 replies
- 4 kudos
- 11234 Views
- 2 replies
- 1 kudos
Reading a table from a catalog that is in a different/external workspace
I am trying to read a table that is hosted on a different workspace. We have been told to establish a connection to said workspace using a table and consume the table.Code I am using isfrom databricks import sqlconnection = sql.connect(server_hostnam...
- 11234 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi Addy Greetings! You can also use Delta sharing to share the data across multiple workspaces. Since you want to read tables from another workspace you can use databricks to databricks delta sharing. https://docs.databricks.com/en/data-sharing/read...
- 1 kudos
- 3875 Views
- 3 replies
- 0 kudos
live spark driver log analysis
In databricks, if we want to see the live log of the exuction we can able to see it from the driver log page of the cluster.But in that we can't able to search by key word instead of that we need to download every one hour log file and live logs are ...
- 3875 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @shan_chandra ,It is like we are putting our driver log into another cloud platform, But here I want to check the live log in local machine tools, is this possible?
- 0 kudos
- 3668 Views
- 5 replies
- 0 kudos
Query Serverless SQL Warehouse from Spark Submit Job
I am trying to load data from a table in SQL warehouse using spark.sql("SELECT * FROM <table>") in a spark submit job, but the job is failing with [TABLE_OR_VIEW_NOT_FOUND] The table or view . The same statement is working in notebook but not in a jo...
- 3668 Views
- 5 replies
- 0 kudos
- 0 kudos
- when you query table manually and running job - do both those actions happens in same Databricks Workspace- what is job configuration - who is job Owner or Run As Account -> do this principal/persona has access to the table ?
- 0 kudos
- 3106 Views
- 2 replies
- 0 kudos
Convert pdf's is into structured data
Is there anything on Databricks to help read PDF (payment invoices and receipts for example) and convert it to structured data?
- 3106 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks! Converting PDF format is sometimes a difficult task as not all converters provide accuracy. I want to share with you one interesting tool I recently discovered that can make your work even more efficient. I recently came across an amazing onl...
- 0 kudos
- 5940 Views
- 3 replies
- 0 kudos
Resolved! Error on Starting Databricks SQL Warehouse Serverless with Instance Profile
I have two workspaces, one in us-west-2 and the other in ap-southeast-1. I have configured the same instance profile for both workspaces. I followed the documentation to set up the instance profile for Databricks SQL Warehouse Serverless by adding th...
- 5940 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Tam , Hope you are doing well! I checked the error in details and it would be because the Instance Profile Name and the Role ARN name don't match exactly. Please see points 3 and 4 here in the docs: https://docs.databricks.com/sql/admin/serverle...
- 0 kudos
- 3137 Views
- 1 replies
- 0 kudos
CDC DLT
Hi all,I would appreciate some clarity regarding the DLT and CDC. So my first question would be, when it comes to the "source" table in the synta, is that CDC table or? Further, if we want to use only databricks, would mounting foreign catalog be a g...
- 3137 Views
- 1 replies
- 0 kudos
- 7664 Views
- 4 replies
- 1 kudos
Rewrite Notebooks Programatically
Hello,I want to refactor the notebook programatically. So, written the code as follows: import requestsimport base64# Databricks Workspace API URLsworkspace_url = f"{host}/api/2.0/workspace"export_url = f"{workspace_url}/export"import_url = f"{worksp...
- 7664 Views
- 4 replies
- 1 kudos
- 2922 Views
- 1 replies
- 0 kudos
Databricks Error while executing this line of code
import geopandas as gpdfrom shapely.geometry import *Pd_csv_sel_pq_gg = gpd.GeoDataFrame(Points_csv_sel_pq_gg.toPandas(), geometry="geometry") Error is given below /databricks/spark/python/pyspark/sql/pandas/utils.py:37: DeprecationWarning: distutil...
- 2922 Views
- 1 replies
- 0 kudos
- 3349 Views
- 2 replies
- 1 kudos
Processing Notebook in python
Hi,I exported notebook from my workspace into my local machine and want to read it in my python code .Is there a way to read the content of my notebook programmatically and make necessary changes and save as dbc/html notebook.
- 3349 Views
- 2 replies
- 1 kudos
- 1 kudos
Not sure what you are trying to accomplish here. If you want to export a notebook as python to do manual editing locally, and then import it back into your workspace why not use repos and connect to it using VSCode etc? You can export the notebook as...
- 1 kudos
- 10716 Views
- 5 replies
- 1 kudos
Dash in Databricks notebook directly
Hi team,Is there a way to embed plotly dash directly inside Databricks notebook?Thanks
- 10716 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @MikeGo - Unfortunately, it's not possible today to embed Dash in a Databricks notebook cell without our Enterprise-level databricks-dash library. Longer term, we are working towards Dash natively working within Databricks notebooks, but that time...
- 1 kudos
- 2386 Views
- 0 replies
- 0 kudos
Foreign Catalog SQL Server Dynamic Port
When creating a Foreign Catalog SQL Server Connection, a port number is required. However, many sql servers have dynamic ports and the port number keeps changing. Is there a solution for this?In most common cases, it should allow instance name instea...
- 2386 Views
- 0 replies
- 0 kudos
- 10575 Views
- 2 replies
- 0 kudos
Spark submit - not reading one of my --py-files arguments
Hi. In Databricks workflows, I submit a spark job (Type = "Spark Submit"), and a bunch of parameters, starting with --py-files.This works where all the files are in the same s3 path, but I get errors when I put a "common" module in a different s3 pat...
- 10575 Views
- 2 replies
- 0 kudos
- 0 kudos
This below is catered for yarn modeif your application code primarily consists of Python files and does not require a separate virtual environment with specific dependencies, you can use the --py-files argument in spark-submitspark-submit --verbose ...
- 0 kudos
- 2245 Views
- 0 replies
- 0 kudos
How to set a job trigger in a yml deployment asset bundle?
Working on an asset bundle/yml file to deploy a job and some notebooks. How to specify within the yml file a trigger to run the job when files arrive at a specified location? Thanks in advance!
- 2245 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 561 | |
| 349 | |
| 287 |
