- 2091 Views
- 2 replies
- 0 kudos
Merge version data files of Delta table
Hi,I am having one CDC enabled Delta table. In 256th version, table is having 50 data files. I want all to merge and create a single file. How can I merge all 50 data file and when I query for 256th version, I should get 1 data file? Is there any com...
- 2091 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, ae you talking about merging CSV files? https://community.databricks.com/t5/machine-learning/merge-12-csv-files-in-databricks/td-p/3551#:~:text=Use%20Union()%20method%20to,from%20the%20specified%20set%2Fs.
- 0 kudos
- 1425 Views
- 0 replies
- 0 kudos
Deleting external table takes 8 hrs
Hi,I am trying to delete the data from the external partitioned table, it has around 3 years of data, and the partition is created on the date column.I am trying to delete each partition first and then the schema of the table, which takes around 8hrs...
- 1425 Views
- 0 replies
- 0 kudos
- 1772 Views
- 0 replies
- 0 kudos
why the code breaks below?
from pyspark.sql import SparkSessionfrom pyspark.ml.regression import LinearRegressionfrom pyspark.ml.feature import VectorAssemblerfrom pyspark.ml.evaluation import RegressionEvaluatorfrom pyspark.ml import Pipelineimport numpy as np# Create a Spark...
- 1772 Views
- 0 replies
- 0 kudos
- 1437 Views
- 2 replies
- 0 kudos
Having trouble with ARC (Automated Record Connector) Python Notebook
I'm trying to use Databricks ARC (Automated Record Connector) and running into an object issue. I assume I'm missing something rather trivial that's not related to ARC. #Databricks Python notebook #CMD1 import AutoLinker from arc.autolinker import A...
- 1437 Views
- 2 replies
- 0 kudos
- 0 kudos
https://www.databricks.com/blog/improving-public-sector-decision-making-simple-automated-record-linking and https://github.com/databricks-industry-solutions/auto-data-linkage#databricks-runtime-requirements
- 0 kudos
- 1344 Views
- 2 replies
- 0 kudos
Delta Sharing CDF API error: "RESOURCE_LIMIT_EXCEEDED"
Hi, When attempting to read a particular version from the Databricks Delta Sharing CDF (Change Data Feed) API, even when that version contains only one data file, an error occurs due to a timeout with following message:"errorCode": "RESOURCE_LIMIT_EX...
- 1344 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi Data_Analytics1Use Optimize on your delta tables. Refer https://docs.databricks.com/en/sql/language-manual/delta-optimize.html
- 0 kudos
- 875 Views
- 1 replies
- 0 kudos
Databricks Pricing Model
Hello Everyone!!I'm new on Databricks Platform & I'm using Databricks for learning purposes.I want to undetstand pricing model of databricks. How Databricks calculates DBU from Compute Type & Instance Type For AWS, Azure & GCP.Can Anyone Explain It.T...
- 875 Views
- 1 replies
- 0 kudos
- 0 kudos
I would like to share the following url https://www.databricks.com/product/pricing/product-pricing/instance-types it will help you to get a price estimation
- 0 kudos
- 801 Views
- 1 replies
- 0 kudos
Databricks certification
I joined two events on 7th Sept,2023 and it was told that by oct 1st week 50% discounted voucher will be given .Also after joining I filled survey by attaching lakehouse fundamentals certificate. Still I didn't get any mail regarding this voucher cou...
- 801 Views
- 1 replies
- 0 kudos
- 0 kudos
@SouvikngTo expedite your request, please list your concern on our ticketing portal. Our support staff would be able to act faster on the resolution (our standard resolution time is 24-48 hours). Thank you for posting your concern on Community!
- 0 kudos
- 1722 Views
- 2 replies
- 0 kudos
Disable personal compute for everyone including workspace admins
If we disable personal compute feature in the account console, it gets deactivated only for non-admin users but still admin users are able to create personal compute clusters. Is there way to restrict to everyone? If not, can you raise a feature requ...
- 1722 Views
- 2 replies
- 0 kudos
- 0 kudos
@DebayanI am talking about the personal compute feature here not the way how clusters are created. If personal compute feature is set to delegate, it should be disable for workspace admin users as well. If this is not supported, it's good to have fea...
- 0 kudos
- 1557 Views
- 1 replies
- 0 kudos
how to upgrade pip associated with the default python
We have a job scheduled and submitted via Airflow to Databricks using api: api/2.0/jobs/runs/submit. Each time the job runs an ephemeral cluster will be launched and during the process a virtual env named: /local_disk0/.ephemeral_nfs/cluster_librarie...
- 1557 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, I got an interesting article on the same. You can follow and let us know if this helps. Please tag @Debayan with your next comment which will notify me!
- 0 kudos
- 754 Views
- 0 replies
- 0 kudos
Is it possible to embed Databricks Workspace plataform in my site?
I want to use databricks in my site, because there're some funcionality the user needs to use in my website, but I want the user to use databricks workspace.. is it possible to embed or do something like that ?
- 754 Views
- 0 replies
- 0 kudos
- 1457 Views
- 1 replies
- 1 kudos
Monitor all Streaming jobs to make sure they are in RUNNING status.
Hi Experts,Is there any way that we can monitor all our Streaming jobs in workspace to make sure they are in "RUNNING" status?I could see there is one option to create a batch job that runs frequently and check the status(through REST API) of all str...
- 1457 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi Experts,Is there any way that we can monitor all our Streaming jobs in workspace to make sure they are in "RUNNING" status?I could see there is one option to create a batch job that runs frequently and check the status(through REST API) of all str...
- 1 kudos
- 2994 Views
- 1 replies
- 1 kudos
Call a workspace notebook from a repository notebook
We have a Databricks workspace with several repositories. We'd like to have a place with shared configuration variables that can be accessed by notebooks in any repository.I created a folder named Shared under the root workspace and in that folder, c...
- 2994 Views
- 1 replies
- 1 kudos
- 1 kudos
We have a Databricks workspace with several repositories. We'd like to have a place with shared configuration variables that can be accessed by notebooks in any repository.I created a folder named Shared under the root workspace and in that folder, c...
- 1 kudos
- 1628 Views
- 1 replies
- 0 kudos
Pyspark API reference
All,I am using Azure Databricks and at times I refer to pyspark API's to interact with data in Azure datalake using python, SQL here https://spark.apache.org/docs/3.5.0/api/python/reference/pyspark.sql/index.htmlDoes databricks website has the list o...
- 1628 Views
- 1 replies
- 0 kudos
- 0 kudos
All,I am using Azure Databricks and at times I refer to pyspark API's to interact with data in Azure datalake using python, SQL here https://spark.apache.org/docs/3.5.0/api/python/reference/pyspark.sql/index.htmlDoes databricks website has the list o...
- 0 kudos
- 1158 Views
- 0 replies
- 0 kudos
Partitioning or Processing : Reading CSV file with size of 5 to 9 GB
Hi Team,Would you please guide me onInstance with 28GB and 8 Cores1. how data bricks reading 5 to 9GB files from BLOB storage ? ( directly loaded full file into one nodes memory )2. howmany tasks will be created based on Core ? how many executors wil...
- 1158 Views
- 0 replies
- 0 kudos
- 1435 Views
- 1 replies
- 2 kudos
The base provider of Delta Sharing Catalog system does not exist.
I have enabled system tables in Databricks by following the procedure mentioned here. The owner of the system catalog is System user. I cannot see the schemas or tables of this catalog. It is showing me the error: The base provider of Delta Sharing C...
- 1435 Views
- 1 replies
- 2 kudos
- 2 kudos
I have already enabled all these schemas using the Databricks CLI command. After enabling, I was able to see all the tables and data inside these schemas. Then I disabled the all the schemas using the CLI command mentioned here. Now, even after re-en...
- 2 kudos
