Hello all,I'm in the process of optimizing my tables and I'm running into a confusing situation. I have a table named "trace_messages_fg_streaming_event". If I navigate to the Databricks catalog, it shows stats:Size: 6.7GB, Files: 464But when I look ...
I am running a notebook to get secret value from GCP Secret Manager. This is working well with Single user Access Mode, however it fail when i use a cluster with Shared Access mode. I have specified the same GCP service account on both of these clust...
Thanks for your response.I am using a cloud service account (same account that was used to create the workspace) on the cluster properties in case of both the single user cluster and on the shared user cluster. This service account has all the necess...
Hi, Im trying to find out what is bottleneck on cluster when running loading process.Scenario: Loading CDC changes from sql server to Raw zone and merge changes into Bronze zone and then merge Bronze to Silver. All is orchestrated in data factory as ...
stdout and stderr looks okay, do you have the log4j to share? You can make a doc out of it and share the doc here.
Hello team, I am using commons(commons-dbcp2) Datasource which supports default connection pooling in Spring Java application (rest services to fetch databricks data via JDBC template).Initially all works fine and can get the data from databricks via...
I am seeing the same issue with hikari. When a pooled connection is created then the databricks cluster is terminated (or restarted), the HikariDataSource retains a stale session handle.Why does connection.isValid() returns true then executing any qu...
I created a Catalog and ownership was assigned to meI created databricks account-group on UC, added my user to this account-group, Assigned ownership of the catalog to this account-group.I deleted the account-groupNow, the catalog ownership is showin...
Hi, In addition to the previous message, you can refer to https://docs.databricks.com/en/data-governance/unity-catalog/manage-privileges/index.html#assign-a-metastore-admin to get more information on metastore and etc.
Hi,I am having one CDC enabled Delta table. In 256th version, table is having 50 data files. I want all to merge and create a single file. How can I merge all 50 data file and when I query for 256th version, I should get 1 data file? Is there any com...
Hi, ae you talking about merging CSV files? https://community.databricks.com/t5/machine-learning/merge-12-csv-files-in-databricks/td-p/3551#:~:text=Use%20Union()%20method%20to,from%20the%20specified%20set%2Fs.
we are migrating out project from on-premise to azure , so on-premise database is the SQL server that we are using and azure data lake gen2 is the storage location where store data currently and so far we are currently validating record count of each...
Hi,I am trying to delete the data from the external partitioned table, it has around 3 years of data, and the partition is created on the date column.I am trying to delete each partition first and then the schema of the table, which takes around 8hrs...
from pyspark.sql import SparkSessionfrom pyspark.ml.regression import LinearRegressionfrom pyspark.ml.feature import VectorAssemblerfrom pyspark.ml.evaluation import RegressionEvaluatorfrom pyspark.ml import Pipelineimport numpy as np# Create a Spark...
I'm trying to use Databricks ARC (Automated Record Connector) and running into an object issue. I assume I'm missing something rather trivial that's not related to ARC. #Databricks Python notebook #CMD1 import AutoLinker from arc.autolinker import A...
https://www.databricks.com/blog/improving-public-sector-decision-making-simple-automated-record-linking and https://github.com/databricks-industry-solutions/auto-data-linkage#databricks-runtime-requirements
Initially, My exam was suspended after 20 mins on me into the exam. I stated that there was some internet issue from my end(however there wasn't any). However, I immediately reached out to Kryterion folks but they responded after 50 minutes and allow...
Adding my username - akhil.inala@tredence.comexam - Databricks Certified Data Engineer Associate Exam
Hi, When attempting to read a particular version from the Databricks Delta Sharing CDF (Change Data Feed) API, even when that version contains only one data file, an error occurs due to a timeout with following message:"errorCode": "RESOURCE_LIMIT_EX...
Hi Data_Analytics1Use Optimize on your delta tables. Refer https://docs.databricks.com/en/sql/language-manual/delta-optimize.html
Hello Everyone!!I'm new on Databricks Platform & I'm using Databricks for learning purposes.I want to undetstand pricing model of databricks. How Databricks calculates DBU from Compute Type & Instance Type For AWS, Azure & GCP.Can Anyone Explain It.T...
I would like to share the following url https://www.databricks.com/product/pricing/product-pricing/instance-types it will help you to get a price estimation
Hello experts,Do you know if there are available metadata/log tables with information on databricks job runs instead of using the Workflows UI?Thank you!
I joined two events on 7th Sept,2023 and it was told that by oct 1st week 50% discounted voucher will be given .Also after joining I filled survey by attaching lakehouse fundamentals certificate. Still I didn't get any mail regarding this voucher cou...
@SouvikngTo expedite your request, please list your concern on our ticketing portal. Our support staff would be able to act faster on the resolution (our standard resolution time is 24-48 hours). Thank you for posting your concern on Community!
