- 1703 Views
- 6 replies
- 1 kudos
Accessing Job parameters using cluster v15.4
After upgrading databricks cluster to version 15.4, is there any way to access job parameters in notebook except the following way ?dbutils.widgets.get("parameter_name")In v15.4, dbutils.notebook.entry_point.getCurrentBindings() has been discontinued...
- 1703 Views
- 6 replies
- 1 kudos
- 1 kudos
For me it is working at 15.4 LTS (includes Apache Spark 3.5.0, Scala 2.12)
- 1 kudos
- 1438 Views
- 2 replies
- 2 kudos
PySparkRuntimeError: [CONTEXT_ONLY_VALID_ON_DRIVER]
I've troubleshot this like 20+ times. I am aware that the current code is causing the spark session to be passed to the workers, where it should only be applied to the driver. Can someone please help me resolve this (the schema is defined earlier)?--...
- 1438 Views
- 2 replies
- 2 kudos
- 2 kudos
You cannot use Spark session explicitly inside Executor logic. Here you are trying mapPartitions which makes the customlogic to get executed inside the executor thread. Either you need to change whole problem approach to segregate spark variable usag...
- 2 kudos
- 4305 Views
- 4 replies
- 0 kudos
Connect snowflake to Databricks
Hey Folks,I just want to know if there is a way to mirror the Snowflake tables in Databricks , Meaning creating a table using format snowflake and give in options of table (host,user,pwd and dbtable in snowflake). I just tried it as per this code bel...
- 4305 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Alberto_Umana , Just a QQ would we be able to change table properties like adding column details, column tagging and Column level masking on the snowflake tables that are under the foreign catalog created?
- 0 kudos
- 21056 Views
- 11 replies
- 15 kudos
Resolved! Get file information while using "Trigger jobs when new files arrive" https://docs.databricks.com/workflows/jobs/file-arrival-triggers.html
I am currently trying to use this feature of "Trigger jobs when new file arrive" in one of my project. I have an s3 bucket in which files are arriving on random days. So I created a job to and set the trigger to "file arrival" type. And within the no...
- 21056 Views
- 11 replies
- 15 kudos
- 15 kudos
Issue with Databricks File Arrival Trigger – Missing File Name InformationThe File Arrival Trigger in Databricks is practically useless if it does not provide the file name and path of the triggering file. In Azure Blob Storage triggers (Function App...
- 15 kudos
- 1705 Views
- 4 replies
- 0 kudos
Resolved! how to read excel files inside a databricks notebook?
Hi community,Is it possible to read excel files from dbfs using a notebook file inside Databricks? If yes, how to do it?
- 1705 Views
- 4 replies
- 0 kudos
- 2622 Views
- 2 replies
- 0 kudos
Resolved! Delta Live Tables - BAD_REQUEST: Pipeline cluster is not reachable.
Hi all,I have very simple pipeline: -- Databricks notebook source CREATE OR REFRESH STREAMING TABLE `catalog-prod`.default.dlt_table AS SELECT * FROM STREAM read_files('/Volumes/catalog-prod/storage/*', format=> 'json') -- COMMAND ---------- CREATE...
- 2622 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi,thank you for your response. In the mean time I found the bug Databricks UI which caused this behaviour. I will raise ticket to Databricks. Please see the draft of the ticket bellow for workaround: We’re facing an issue with Delta Live Tables pip...
- 0 kudos
- 1089 Views
- 1 replies
- 0 kudos
Using databricks-sql-connector in Notebook
I am attempting to utilse the databricks-sql-connector python package within a generalised application deployed to run within a Databricks notebook. Upon attempting to import, I am receiving a module not found error. However, the package is visible ...
- 1089 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @wilmorlserios The import it's incorrect. It should be: from databricks import sql
- 0 kudos
- 18754 Views
- 4 replies
- 1 kudos
ModuleNotFoundError: No module named 'pyspark.dbutils'
I have a class in a python file like this from pyspark.sql import SparkSession from pyspark.dbutils import DBUtils class DatabricksUtils: def __init__(self‌‌): self.spark = SparkSession.getActiveSession() self.dbutils = DBUtil...
- 18754 Views
- 4 replies
- 1 kudos
- 1 kudos
Had the same Problem in my GitLab CI/CD Pipeline while trying to deploy: $ databricks bundle deploy -t dev Building package... Error: build failed package, error: exit status 1, output: Traceback (most recent call last): [...] File "/builds/user/...
- 1 kudos
- 5799 Views
- 5 replies
- 1 kudos
Connecting from Databricks to Network Path
Hi All,Will appreciate if someone can help me with some references links on connecting from Databricks to external network path. I have Databricks on AWS and previously used to connect to files on external network path using Mount method. Now Databri...
- 5799 Views
- 5 replies
- 1 kudos
- 1058 Views
- 3 replies
- 0 kudos
Databrick select from web address that returns JSON
Hi I'm in a data bricks notebook and want to select from a web site that returns json. For example this web site http://ergast.com/api/f1/2004/1/results.jsonwill return some JSON. (example only) Can i do the following or Do you need to use python. (...
- 1058 Views
- 3 replies
- 0 kudos
- 0 kudos
yes, you could use SQL to read the data from the volume:%sql SELECT * FROM json.`/Volumes/demo/raw/files/ergast/my_results.json`You still have to flatten the result.The thing with the shell was just an example. Wouldn't recommend that, just use pytho...
- 0 kudos
- 7717 Views
- 7 replies
- 8 kudos
Databricks Jobs and CICD
Hi, We currently leverage Azure DevOps to source control our notebooks and use CICD to publish the notebooks to different environments and this works very well. We do not have the same functionality available for Databricks jobs (the ability to sourc...
- 7717 Views
- 7 replies
- 8 kudos
- 8 kudos
To manage Databricks jobs within a DevOps pipeline, start by exporting the job configuration as a JSON file from the Databricks workspace. Parameterize this JSON by replacing environment-specific values with placeholders. Integrate the parameterized ...
- 8 kudos
- 1281 Views
- 1 replies
- 1 kudos
I attended the Advanced Machine Learning with Databricks training last week virtually I am still unable to get the day 2 session videos of any of the...
I attended the Advanced Machine Learning with Databricks training last week virtually I am still unable to get the day 2 session videos of any of the Instructor led Paid Trainings. They are supposed to be available for replay with in 24 hours but I ...
- 1281 Views
- 1 replies
- 1 kudos
- 1 kudos
I have the same problem.
- 1 kudos
- 1088 Views
- 2 replies
- 0 kudos
Data processing metrics
Dear all,What are some proven ways of capturing data processing metrics (number of rows processed/updated/inserted, number of micro-batches etc etc) in a PySpark/SQL code based notebook irrespective of the fact it uses auto-loader, structured streami...
- 1088 Views
- 2 replies
- 0 kudos
- 0 kudos
@Alberto_Umana thanks for the reply. without the current code change, is it possible to extract these metrics from logs?
- 0 kudos
- 2975 Views
- 10 replies
- 1 kudos
My DABS CLI Deploy call not generating a .tfstate file
Hi Community,I'm running into an issue, when executing Databricks CLI Bundle Deploy I dont get the Terraform State file (.tfstate). I know that I should get one but even when defining the state_apth on my YAML (.yml) DABS file I still do not get it.D...
- 2975 Views
- 10 replies
- 1 kudos
- 5219 Views
- 4 replies
- 2 kudos
[AWS] How do you replace the Account Admin?
I need to remove an older admin that previously set up the Databricks Account. However, I get an error (even through I am also an Account Admin).How do I replace a prior account admin? Or at least remove their admin status and/or disable the accoun...
- 5219 Views
- 4 replies
- 2 kudos
- 2 kudos
Was this ever fixed as I have got the exact same issue as our original AWS Account Admin has left the business and I now cannot remove his Account Admin role (but I have disabled access etc)!
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 554 | |
| 349 | |
| 287 |
