- 2395 Views
- 1 replies
- 0 kudos
how to upgrade pip associated with the default python
We have a job scheduled and submitted via Airflow to Databricks using api: api/2.0/jobs/runs/submit. Each time the job runs an ephemeral cluster will be launched and during the process a virtual env named: /local_disk0/.ephemeral_nfs/cluster_librarie...
- 2395 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, I got an interesting article on the same. You can follow and let us know if this helps. Please tag @Debayan with your next comment which will notify me!
- 0 kudos
- 3323 Views
- 1 replies
- 1 kudos
How connect connection ORACLE onpremise in Databricks for extract data
Hello, How Connect in Databricks Enterprise connection Oracle Onpremise and that permissions is necessary.Thank you
- 3323 Views
- 1 replies
- 1 kudos
- 1680 Views
- 0 replies
- 0 kudos
Using map_filter / filter to return only map elements for a given predicate
I want to apply a filter to a map structure (on a column called "ActivityMap") for elements only where a given predicate holds. Representative data is below. Applying both "map_filter" and "array_contains" will return rows where a predicate holds, ho...
- 1680 Views
- 0 replies
- 0 kudos
- 971 Views
- 0 replies
- 0 kudos
Is it possible to embed Databricks Workspace plataform in my site?
I want to use databricks in my site, because there're some funcionality the user needs to use in my website, but I want the user to use databricks workspace.. is it possible to embed or do something like that ?
- 971 Views
- 0 replies
- 0 kudos
- 1734 Views
- 0 replies
- 0 kudos
Questions about the new Serverless SQL
I have a list of questions about the Serverless SQL option:As per the docs, it says it's hosted by Databricks. Is there any exception to that? Will it ever create ec2 instances on AWS for the serverless option?Are any serverless asset stored on custo...
- 1734 Views
- 0 replies
- 0 kudos
- 1975 Views
- 1 replies
- 1 kudos
Monitor all Streaming jobs to make sure they are in RUNNING status.
Hi Experts,Is there any way that we can monitor all our Streaming jobs in workspace to make sure they are in "RUNNING" status?I could see there is one option to create a batch job that runs frequently and check the status(through REST API) of all str...
- 1975 Views
- 1 replies
- 1 kudos
- 5711 Views
- 2 replies
- 4 kudos
Can we apply multiple exam discount voucher to one exam?
I have a 75% off certification exam voucher using my working email, and i also noticed that you can redeem a 25% off voucher from the rewards store. I am wondering if you can add these 2 vouchers together at the same time so that you can get 1 exam 1...
- 5711 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi chenqian0562,Could you please share the solution with the community? I have the same issue.
- 4 kudos
- 1749 Views
- 0 replies
- 0 kudos
How can I resolve a problem between AWS and Databricks platform?
This is Algograp Co., Ltd., a partner of Databricks.There was a problem in the process of subscribing to Databricks and linking AWS account.I had a problem using my existing platform and canceled my subscription. Account activation is not possible du...
- 1749 Views
- 0 replies
- 0 kudos
- 6682 Views
- 0 replies
- 0 kudos
DLT use case
Is Delta Live Tables (DLT) appropriate for data that is in the millions of rows and GB sized? Or is DLT only optimal for larger data with billions of rows and TB sized?Please consider the Total Cost of Ownership.Development costs (engineering time)O...
- 6682 Views
- 0 replies
- 0 kudos
- 3784 Views
- 1 replies
- 1 kudos
Call a workspace notebook from a repository notebook
We have a Databricks workspace with several repositories. We'd like to have a place with shared configuration variables that can be accessed by notebooks in any repository.I created a folder named Shared under the root workspace and in that folder, c...
- 3784 Views
- 1 replies
- 1 kudos
- 1940 Views
- 1 replies
- 0 kudos
Pyspark API reference
All,I am using Azure Databricks and at times I refer to pyspark API's to interact with data in Azure datalake using python, SQL here https://spark.apache.org/docs/3.5.0/api/python/reference/pyspark.sql/index.htmlDoes databricks website has the list o...
- 1940 Views
- 1 replies
- 0 kudos
- 1527 Views
- 0 replies
- 0 kudos
Partitioning or Processing : Reading CSV file with size of 5 to 9 GB
Hi Team,Would you please guide me onInstance with 28GB and 8 Cores1. how data bricks reading 5 to 9GB files from BLOB storage ? ( directly loaded full file into one nodes memory )2. howmany tasks will be created based on Core ? how many executors wil...
- 1527 Views
- 0 replies
- 0 kudos
- 7254 Views
- 1 replies
- 1 kudos
Set default database thru Cluster Spark Configuration
Set the default catalog (AKA default SQL Database) in a Cluster's Spark configuration. I've tried the following :spark.catalog.setCurrentDatabase("cbp_reporting_gold_preprod") - this works in a Notebook but doesn't do anything in the Cluster.spark.sq...
- 7254 Views
- 1 replies
- 1 kudos
- 1 kudos
I've tried different commands in the Cluster's Spark Config, none work, they execute at Cluster startup w/o any errors shown in the logs, but once you run a notebook attached to the cluster Default catalog is still set to 'default'.
- 1 kudos
- 4011 Views
- 4 replies
- 1 kudos
Resolved! My exam has suspended by an unprofessional proctor- need help to reschedule
case #00378268 I passed Spark Developer associate exam about 6 month ago with great experience. However, this time the proctor did not even bother to show up to start the exam - checking ID, the room and the surroundings. Somehow, I was able to st...
- 4011 Views
- 4 replies
- 1 kudos
- 1934 Views
- 1 replies
- 2 kudos
The base provider of Delta Sharing Catalog system does not exist.
I have enabled system tables in Databricks by following the procedure mentioned here. The owner of the system catalog is System user. I cannot see the schemas or tables of this catalog. It is showing me the error: The base provider of Delta Sharing C...
- 1934 Views
- 1 replies
- 2 kudos
- 2 kudos
I have already enabled all these schemas using the Databricks CLI command. After enabling, I was able to see all the tables and data inside these schemas. Then I disabled the all the schemas using the CLI command mentioned here. Now, even after re-en...
- 2 kudos
